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Description 

BACKGROUND OF THE INVENTION 

1 . Ftekj of the invention: s 

[0001] The present invention relates to the field of 
graphical user Interfaces for computer systems and, 
more particularly, to a method and apparatus for provid- 
ing an Intuitive methodology for an operator to control to 
remote devices with a computer controlled object-ori- 
ented user interface utilizing animated graphic images. 

2. Art Background: 

13 

[0002] Most popular graphical user interfaces that are 
available today, such as the Xerox Star®, Apple Macin- 
tosh®, and Microsoft Windows graphical user interfac- 
es, are based upon the "desktop metaphor." In a desk- 
top metaphor system, the display screen is treated as a *° 
virtual desktop. Graphical symbols placed on the virtual 
desktop (i.e. the display screen) are used to represent 
common objects found in an office environment such as 
files, fUe folders, and printers. These desktop metaphor 
systems have several limitations which hamper the ef- *5 
fectiveness of the man-machine interface between the 
human operator and the computer system. 
[0003] One problem with the existing desktop meta- 
phor graphical user interfaces Is the use of a hierarchical 
file system. Most desktop metaphor graphical user in- 90 
terfaces are based upon a computer having a file system 
in which the computer files are organized in a hierarchi- 
cal structure. The hierarchical file system structure has 
a root directory and several sub-directories within the 
root directory. Each sub-directory may have sub-sub-di- 35 
rectories, and so on. The hierarchical file system struc- 
ture is graphically illustrated on the screen as some type 
of root file cabinet with file folders inside. The Microsoft 
Windows and Apple Macintosh® operating systems 
both use hierarchical file systems illustrated as file fold- 
ers. 

[0004] The hierarchical structure of file folders is often 
very confusing to new users. For example, many users 
get lost while navigating through hierarchical file sys- 
tems because the system of keeping files in fUe folders, 45 
which are in other file folders nested to an arbitrary depth 
is inherently confusing. Users become lost because 
each file folder looks like every other file folder, and the 
user Is provided with Httie feedback as to his location 
within the file system hierarchy. Also, each file folder so 
provides no Information as to what the file folder con- 
tains other than a user-defined label. For example, ex- 
cept for a short text label, which may have been defined 
by another user, the user generally is not able to deter- 
mine whether an open folder is an accounting file folder 55 
containing spreadsheet data, or a data base file folder 
containing a customer data base. Therefore, navigating 
through the complex file system hierarchy Is difficult 



since the current desktop metaphor graphical user in- 
terface offers little feedback as to a user's location within 
the file hierarchy. 

[0005] In current graphical user interface systems 
based upon the desktop metaphor, certain applications 
often do not fit well into a desktop metaphor environment 
conceptually. Specifically, non-office related applica- 
tions may be Inappropriate for a system based upon the 
desktop metaphor. For example, in a desktop metaphor, 
an application that controls a television would have to 
be placed into a "paper" file folder on the desktop. 
[0006] Another common problem with desktop meta- 
phor user interfaces is that displaying a large number of 
applications on the desktop results in a cluttered desk- 
top. Many users of desktop metaphor systems keep 
large numbers of applications and files, usually in icon 
form, on the display screen. Some attempt is made to 
use a graphical representation that identifies the type of 
application or file. However, when a user wishes to open 
a particular application or file, the user must search the 
display screen to find the desired icon just as if the user 
were trying to find a particular paper file on a messy 
desktop. Icons placed the desktop have no apparent hi- 
erarchy, and are ail equal. This equality may confuse a 
user, and may create problems when the user wants to 
switch from one task to another. 
[0007] Several systems have evolved to Improve the 
organization of the desktop metaphor graphical user in- 
terface. One system is the concept of "Rooms". See, 
Patrick Chan, 'Learning Considerations In User Inter- 
face Design: The Room ModeT, Report CS-84-1 6, Uni- 
versity of Waterloo Computer Science Department, On- 
tario, Canada, July, 1984; Brenda Laurel, "Computers 
as Theatre", Addison-Wesley Publishing Co., 1991; 
Brenda Laurel, "The Art of Human-Computer Interface 
Design", Addison-Wesley Publishing Co., 1990; U.S. 
Patent 5,072, 41 2, Henderson, Jr. et al. , User Interface 
with Multiple Workspaces for Sharing Display System 
Objects', and the references cited therein. 
[0008] In systems based upon the Rooms concept, 
the user creates different rooms where the user per- 
forms a single task or a group of related tasks. Each 
room represents an individual virtual desktop with a 
group of associated tasks. Each room Is given a label 
that identifies the room with the room's functions. For 
example, a user may create a mail room, a typing room 
with a word processor, and an accounting room with a 
spreadsheet Each room has only those icons and win- 
dows that are associated with the particular task of the 
mom, as well as any windows shared by all rooms, such 
as a dock. 

[0009] To switch between tasks in a system that uses 
the Rooms concept, a user moves between the different 
rooms that are associated with the different tasks. For 
example, to go from a spreadsheet to retrieve email, a 
user moves from the accounting room to the mail room. 
To move between the two rooms, a user selects an icon 
that represents a door connecting the current room to 
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the other room. Users can create doors between any of 
the available rooms. 

[001 OJ Although the Rooms concept helps solve the 
problem of the "cluttered desktop, ■ the Rooms concept 
still relies upon the common desktop metaphor. For ex* 5 
ample, the Rooms interface concept does not help sim- 
plify the confusing hierarchical file structure used to 
store information. The Rooms concept also does little to 
solve the form and function inconsistencies that are ap- 
parent when applications which do not fit into the desk- 10 
top metaphor are added to the system. 
[001 1] In prior art graphical user Interfaces, each time 
a new piece of hardware is introduced to the system a 
new piece of control software must also be added by the 
user. For example, If a user of an Apple® Macintosh® u 
or a user of Microsoft® Windows desires to add a new 
printer to the system, the user must also install a new 
printer driver to control the added printer. Therefore, 
each time a user desires to add a new device to the sys- 
tem, the user must also follow a complicated procedure 20 
to add the appropriate software. 
[001 2] As will be described, the present invention pro- 
vides an improved graphical user interface that over- 
comes several of the problems associated with desktop 
metaphor systems. A method and apparatus for obtain- 25 
ing the user interface of a remote device without requir- 
ing the user to load the related software is also dis- 
closed. The remote device can be controlled using the 
remote device's user interface as displayed on the 
present invention's display screen. Although the user in- &> 
terface Is described with reference to the control of re- 
mote devices, it will be appreciated from the discussion 
that follows that the user interface may be applied to a 
variety of applications In computer workstations, porta- 
ble computers, and hand held control systems. 35 
[001 3] Further examples of various prior art arrange- 
ments are described in EP-a-0439873 (IBM et al) and 
IBM Technical Disclosure Bulletin, vol. 36, No. 4, April 
1993, New York, US, pages. 215-216, XP002012167, 
"Animated Command Execution Icon". *o 

SUMMARY OF THE INVENTION 

[0014] The present invention provides an intuitive 
graphical user interface and a method for controlling re- 45 
mote devices as set forth in the appended claims. 
[0015] The graphical user interface is displayed on a 
hand-held display device. The hand-held display device 
comprises at least one central processing unit (CPU) 
coupled to a graphic display system. The graphic dis- so 
play system Is touch sensitive to permit the user to Input 
information using a fingertip without the need for addi- 
tional input devices such as a keyboard. A communica- 
tions circuit is also coupled to the CPU transmitting in- 
formation over some communication medium. The com- « 
munications circuit is used to control and to communi- 
cate with remote devices. 

[001 6] The graphical user Interface creates a world of 



spaces organised Into a geographic map structure that 
is simple to navigate through since each space provides 
familiar geographic surrounds. Each space is represent- 
ed by a background image which provides navigational 
landmarks thai Identify where the space is located within 
the geographic map structure. For example, the back- 
ground Image of a living room space contains Items typ- 
ically found In a living room such as a lamp, a chair, and 
a table. Therefore, navigation through this "world" is 
made easier because the user can orient himself by 
looking around and examining the surroundings Just as 
a person would do In the real world. 
[0017] Within each Space, the user interface provides 
graphic objects that can be selected and manipulated 
by the user. The graphic objects are rendered as ani- 
mated cartoon-like graphic objects that are easily iden- 
tifiable as real world objects such as televisions and 
VCRs. There are at least three types of objects that exist 
in the Spaces of the user Interface: data objects, but- 
tons, and portals. Data objects are objects that are as- 
sociated with a particular piece of data. Buttons are ob- 
jects that are associated with a particular function or ac- 
tion; when a user selects a button, the user interface 
performs the function or action associated with the but- 
ton. Portals are objects that are displayed in a first 
Space but are associated with a second Space; when a 
user selects a portal the user passes through the portal 
and moves to the second Space associated with that 
portal. The user environment is further enhanced by the 
use of sound, coordinated with every user action and 
supplementing the visual feedback provided. 
[001 8] To control real world remote devices, many of 
the graphic objects presented on the screen are asso- 
ciated with remote devices In the real world. To control 
a remote device, a user selects the graphic object pic- 
tured on the display screen associated with the particu- 
lar remote device. After the user selects a graphic object 
associated with a remote device, the hand-held display 
device displays a user Interface for controlling the re- 
mote device. The user can then interact with the remote 
device's user Interface. To control the remote device, the 
present invention invokes a device driver method for the 
remote device. The device driver method may be locat- 
ed within the remote device or within the local hand-held 
display device, depending on the remote device's so- 
phistication. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0019] The objects, features and advantages of the 
present Invention will be apparent to one skilled In the 
art, in view of the following detailed description in which: 

FIGURE 1a illustrates an overview of the remote 
control functionality provided by the present Inven- 
tion. 

FIGURE 1 b illustrates an overview of the hardware 
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used to Implement the hand-held display device of 
the present invention. 

FIGURE 1c illustrates a block diagram of an intelli- 
gent remote device that is controlled by the present 5 
Invention. 

FIGURE 1d illustrates a block diagram of a simple 
remote device that Is controlled by the present in- 
vention. 10 

FIGURE 1e illustrates a block diagram of conven- 
tional electronic device that is controlled remotely 
by the present Invention. 

15 

FIGURE 2a Illustrates a structure of different geo- 
graphic spaces. 

FIGURES 2b - 2c illustrate a sample geographic 
map structure of spaces used to provide a user in- 20 
terface. 

FIGURE 3 illustrates a view of the outer-space 
Space of the user Interface based upon the geo- 
graphic map structure in Figures 2b and 2c. 25 

FIGURE 4 Illustrates a first view of the Earth's sur- 
face Space in the user Interface based upon the ge- 
ographic map structure in Figures 2b and 2c. 

30 

FIGURE 5 illustrates a second view of the Earth's 
surface Space in the user interface based upon the 
geographic map structure in Figures 2b and 2c. 

FIGURE 6 illustrates a view of the first house Space S5 
of the user Interface based upon the geographic 
map structure in Figures 2b and 2c. . 

FIGURE 7 illustrates a view of a Irving room Space 
in the user interface based upon the geographic *o 
map structure in Figures 2b and 2c. 

FIGURE 8 illustrates a view of a television Space in 
the user interface based upon the geographic map 
structure in Figures 2b and 2c. 45 

FIGURE 9 illustrates a view of a television program 
listing book Space in the user interface based upon 
the geographic map structure in Figures 2b and 2c. 

so 

FIGURE 10 Illustrates a view of a single television 
program description Space of the user interface 
based upon the geographic map structure in Fig- 
ures 2b and 2c. 

55 

FIGURE 1 1 illustrates a conceptual view of the tel- 
evision program listing book Space of Figure 9. 



FIGURE 12 illustrates a view of a plurality of objects 
which do not fit on a single screen display. 

FIGURE 1 3 illustrates a conceptual view of the plu- 
rality of objects placed on an object wheel. 

FIGURE 14 Illustrates a conceptual view of the ob- 
ject wheel of Figure 13 where the object wheel has 
rotated revealing new objects. 

FIGURE 15 Illustrates the Spin gesture being used 
to spin an object wheel. 

FIGURE 16s illustrates a Pickup gesture being 
used to pickup an object off an object wheel. 

FIGURE 16b illustrates a Pickup gesture being 
used to pickup an object from a Space. 

FIGURE 17s illustrates the Agent Portal of the 
present invention wearing a doctor's uniform to 
symbolize a diagnostic mode. 

FIGURE 17b illustrates the Agent Portal of the 
present invention holding a pencil to symbolize a 
writing mode. 

FIGURE 17c illustrates the Agent Portal of the 
present invention attempting to alert the user about 
a recent event 

FIGURE 18a illustrates an Intelligent device trans- 
mitting its graphical user interface to a display de- 
vice. 

FIGURE 18b Illustrates a display device displaying 
the graphical user interface of an intelligent control- 
lable device. 

RGURE 18c illustrates the display device transmit- 
ting user Interactions with the user interface to the 
intelligent device. 

RGURE 18d illustrates the intelligent device trans- 
mitting return values or feedback to the display de- 
vice. 

RGURE 19a illustrates a conceptual diagram of the 
program objects In the intelligent device being sent 
to the display device. 

RGURE 19b illustrates a conceptual diagram of the 
program objects of the intelligent device in the dis- 
play device. 

RGURE 19c illustrates a conceptual diagram of the 
display device invoking a method in an object within 
the remote Intelligent device. 
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FIGURE 1 9d illustrates a conceptual diagram of the 
method In an object within the remote intelligent de- 
vice returning status. 

FIGURE 20 illustrates a flowchart of the transfer of 5 
an Intelligent device's user interface. 

FIGURE 21a illustrates a state diagram of the RPC 
manager for a client requesting a remote method 
Invocation. io 

FIGURE 21 b Illustrates a state diagram of the RPC 
manager for a server handling a remote method in- 
vocation. 

13 

FIGURE 21c illustrates the sequence of a remote 
method Invocation. 

FIGURE 22a illustrates the simple device transfer- 
ring Its graphical user Interface to a display device. 

RGURE 22b illustrates a display device displaying 
the graphical user interface of the simple controlla- 
ble device. 

25 

RGURE 22c illustrates the display device transmit- 
ting simple control codes to the simple device. 

RGURE 22d illustrates the simple device transmit- 
ting return values or feedback to the display device. & 

RGURE 23a illustrates a conceptual diagram of the 
program objects in a simple device being sent to the 
display device. 

35 

FIGURE 23b illustrates a conceptual diagram of the 
program objects of the simple device in the display 
device. 

RGU RE 23c illustrates a conceptual diagram of the 4 <> 
display device Invoking a method in a program ob- 
ject and transmitting control codes to the simple de- 
vice. 

RGURE 23d illustrates a conceptual diagram of the 
remote simple device returning status information. 



convey the substance of their work to others skilled In 
the art. 

[0021 ] An algorithm is here, and generally, conceived 
to be a consistent sequence of steps leading to a desired 
result. These steps are those requiring physical manip- 
ulations of physical quantities. Usually, though not nec- 
essarily, these quantities take the form of electrical or 
magnetic signals capable of being stored, transferred, 
combined, compared, selected, chosen, modified, and 
otherwise manipulated. It proves convenient at times, 
principally for reasons of common usage, to refer to 
these signals as bits, values, elements, symbols, char- 
acters, Images, terms, numbers, or the like. It should be 
borne in mind, however, that all of these, as well as sim- 
ilar terms, are to be associated with the appropriate 
physical q uantrties and are merely convenient labels ap- 
plied to these quantities. 

[0022] In the present case, the operations are ma- 
chine operations performed in conjunction with a human 
operator. Useful machines for performing the operations 
of the present invention include general purpose digital 
computers or other similar devices. In ail cases, the dis- 
tinction between the method operations of operating a 
computer and the method of computation Itself should 
be kept in mind. The present invention relates to method 
steps for operating a computer and processing electrical 
or other physical signals to generate other desired phys- 
ical signals. 

[0023] The present Invention also relates to appara- 
tus for performing these operations. This apparatus may 
be specially constructed for the required purposes or it 
may comprise a general purpose computer selectively 
activated or reconfigured by a computer program stored 
in the computer. The algorithms presented herein are 
not inherently related to any particular computer or other 
apparatus. In particular, various general purpose ma- 
chines may be used with programs in accordance with 
the teachings herein, or It may prove more convenient 
to construct more specialized apparatus to perform the 
required method steps. The required structure for a va- 
riety of these machines will appear from the description 
given below. Machines which may perform the functions 
of the present invention include those manufactured by 
the Assignee, First Person, Inc., as well as other manu- 
facturers of computer systems, including Sun Microsys- 
tems, Inc. of Mountain View, California. 



RGURE 24 illustrates a flowchart of the transfer of 
a simple device's user interface. 

NOTATION AND NOMENCLATURE 

[0020] The detailed descriptions which follow are pre- 
sented largely in terms of display images, algorithms, 
and symbolic representations of operations of data bits 
within a computer memory. These algorithmic descrip- 
tions and representations are the means used by those 
skilled in the data processing arts to most effectively 



CODING DETAILS 

so [0024] No particular programming language has been 
Indicated for carrying out the various procedures de- 
scribed herein. This Is due in part to the fact that not all 
languages that might be mentioned are universally 
available. Each user of a particular computer will be 

55 aware of a language which is most suitable for his Im- 
mediate purposes. In practice, ft has proven useful to 
substantially implement the present invention in a high 
level language which is then complied Into machine ex- 
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ecutable object code. Because the computers and the 
monitor systems which may be used In practicing the 
instant Invention consist of many diverse elements, no 
detailed program listing has been provided. It is consid- 
ered that the operations and other procedures de- 
scribed herein and illustrated In the accompanying 
drawings are sufficiently disclosed to permit one of or- 
dinary skill to practice the instant invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0025] A computer based remote control system with 
an intuitive graphical user interface Is disclosed. In the 
following description, for purposes of explanation, spe- 
cific nomenclature Is set forth to provide a thorough un- 
derstanding of the present invention. However, it will be 
apparent to one skilled in the art that these specific de- 
tails are not required to practice the present invention. 
In other instances, well known circuits, functions, proc- 
esses, and devices are shown In block diagram and con- 
ceptual diagram form in order not to obscure the present 
Invention unnecessarily. 

System Overview 

[0026] Figure 1a illustrates an overview of the com- 
puter based remote control system of the present Inven- 
tion. The main component of the present invention con- 
sists of a hand-held display device 170 as illustrated in 
Figure 1a. In the present embodiment, the hand-held 
display device 170 comprises a computer system with 
a color graphic touch screen display. The hand-held dis- 
play device 170 is designed to control any compatible 
remote device such as thermostat 150, video cassette 
recorder 39, and stereo system 152. The hand-held dis- 
play device 170 communicates with the remote devices 
using a communications medium 160. 
[0027] The remote control system of the present In- 
vention as illustrated In Figure 1a differs significantly 
from prior art remote control systems since it Is based 
upon modular object-oriented software. The object-ori- 
ented software provides features previously not availa- 
ble in prior art remote control systems. For example, 
when an operator (sometime referred to herein as a "us- 
er") adds a new compatible remote device, such as ther- 
mostat 150 to the system, the new remote device trans- 
mits Information about Itself to the handheld display de- 
vice 170. The transmitted information includes a user 
Interface object that can be used to render a graphical 
user interface for the remote device. The present Inven- 
tion uses the user interface object to display a graphical 
user interface for the new remote device on the hand- 
held display device 170. The transmitted information 
may also include a device driver object that is used to 
control the new remote device. To organize all the avail- 
able user interface objects, the hand-held display device 
170 creates a "world" of geographic Spaces that can be 
easily be navigated using a set of gestures. 



26635 B1 10 

[0028] This Specification will first disclose the hard- 
ware required to construct the hand-held display device 
170 and the compatible remote devices of the present 
invention. The graphical user interface comprising geo- 

5 graphic Spaces and objects which Is displayed on the . 
hand-held display device 170 win then be described. A 
set of gestures used by operators to navigate around 
the user Interface of the present invention wiN also be 
defined. To combine the user interface elements, a tour 

10 through an embodiment of the user Interface of the 
present invention will be provided to Illustrate how the 
navigational gestures are used to interact with the ob- 
jects displayed on the hand-held display device 170. Fi- 
nally, the system of obtaining Information from the re- 

13 mote devices is defined, such that the hand-held display 
device 170 may communicate with and control remote 
external devices. 

[0029] Due to the limitations of a written Specification 
to describe animated events on an Interface, the reader 

20 is referred to a video tape entitled "Green Project User 
Interface," submitted by the applicant concurrently with 
the filing of the application on which this patent is based. 
In addition it must also be noted that the features of the 
present invention are Illustrated in black and white within 

25 the accompanying figures. However, In the presently 
preferred embodiment, and as described below, objects 
and features are displayed In color. 

Hardware Configuration 

90 

[0030] The hand-held display device 170 of the 
present invention comprises a computer system, such 
as that illustrated in Figure 1b. The computer system 
170 of Figure 1b includes a central processing unit 

35 (CPU) 30, a memory unit 33, and an input/output (I/O) 
unit 34. These elements are those typically found in 
most general purpose computers, and in fact, the hand- 
held computer system 170 is intended to be represent- 
ative of a broad category of data processing devices ca- 

40 pable of generating graphic displays. 

[0031] Coupled to the Input/output (I/O) unit 34 of 
computer system 170 Is a color graphic screen display 
37. The screen display 37 comprises, In part, a color 
raster display device such as a cathode ray tube (CRT) 

45 monitor or a liquid crystal display (LCD). The screen dis- 
play 37 must be of sufficient resolution such that screen 
display 37 can render graphic images. 
[0032] In the preferred embodiment, the screen dis- 
play 37 further comprises a touch screen display sys- 

50 tern. The touch screen display Includes a feature that 
can detect the presence of a finger touching the display 
screen 37. Furthermore, the touch screen can identify 
the precise location of the finger touching the display 
screen 37 such that the display screen can sense finger 

55 gestures made by a user's finger on the display screen 
37. The touch screen display 37 may comprise one of 
a variety of touch sensitive display screens commercial- 
ly available on the market 
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[0033] Alternatively, the hand-held computer system 
170 may be constructed with a conventional graphic dis- 
play system without a touch sensitive display screen. 
Such a system requires an alternative input system to 
obtain the user's interactions. Figure 1b Illustrates an 
alternative Input device 42. Although the user interface 
of the present Invention may be used with alternative 
Input devices 42 such as mice, trackballs, and Joysticks, 
the system will be described within this specification as 
the preferred embodiment with a touch screen display 
37. 

[0034] In operation, the CPU 30 generates and dis- 
plays a graphical user interface on touch screen display 
37 through the execution of computer code stored In 
memory 33. The computer code stored in memory 33 is 
not disclosed herein since the description of the present 
invention In this Specification is sufficient for one skilled 
in the computer art to utilize the teachings of the inven- 
tion in a variety of computer systems using one of many 
computer languages. 

[0035] Coupled to the input/output (I/O) unit 34 of 
computer system 170, in Figure 1b, Is an external com- 
munication circuit 35. The external communication cir- 
cuit 35 provides a system for transmitting and receiving 
information to and from the hand-held computer system 
170. The present Invention uses the external communi- 
cation circuit 35 In computer system 170 to communi- 
cate with remote devices such as a video cassette re- 
corder 39 and a thermostat 150, across communica- 
tions medium 160. The communication circuit 35 may 
use one of many well-known communication technolo- 
gies such as radio waves, low power, cellular, infrared 
signals, or data signals carried on existing power lines. 
[0036] To be controlled by the present hand-held dis- 
play device 170 of the present invention, a remote de- 
vice must have at least one method for receiving infor- 
mation transmitted by the hand-held display device 1 70. 
Ideally, the remote devices are intelligent devices that 
can both transmit and receive information. However, the 
hand-held display device 170 can control any remote 
device that fits within any of the three following remote 
device "types". 

[0037] The first type of remote device that can be con- 
trolled by the present invention is an "intelligent remote 
device" specifically designed to operate with the hand- 
held display device 170. In the present Invention, an "In- 
telligent remote device" is defined as a remote device 
that contains a microprocessor, a communication sys- 
tem, and a memory. Figure 1 c provides a block diagram 
of the components in an intelligent remote device 150. 
As illustrated in Figure 1c, the Intelligent remote device 
150 comprises a CPU 403, a memory 404, a communi- 
cations circuit 401 , and a device control circuitry 405. 
[0036] The hand-held display device 170 of the 
present invention communicates with an intelligent re- 
mote device 150 through the Intelligent remote device's 
communications circuit 401. The memory 404 stores 
programs and data used by the CPU 403, including a 
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user Interface object 303 and a device driver object 301 . 
The CPU 403 in the Intelligent remote device invokes 
methods In the device driver object 301 to control the 
remote device. 

5 [0039] The second type of remote device that can be . 
controlled by the present Invention Is a "simple remote 
device" specifically designed to operate with the hand- 
held display device 170. In the present invention, a re- 
mote device is defined as a "simple remote device" if 

10 the remote device does not have a microprocessor or 
the remote device contains such a primitive processor 
that the hand-held display device 170 can not invoke 
methods in a device driver object within the remote de- 
vice. 

15 [0040] Figure 1d provides a block diagram of the 
components in a typical simple remote device 155. As 
illustrated In Figure 1d, the simple remote device 155 
comprises an emitter 420, a receiver 421, a memory 
424, and device control circuitry 425. The emitter 420 in 

20 the simple remote device 155 is used to transmit the 
contents of the memory 424 to the hand-held display 
device 170. Therefore, the simple remote device 155 
transmits both the user interface object 353 and the de- 
vice driver object 351 in the memory 424, such that the 

25 hand-held display device receives both objects. 

[0041] To control the simple remote device 155, the 
display device 170 invokes a method within the device 
driver object 351 to translate the user's interactions into 
simple predefined control codes. The display device 170 

90 then transmits the simple predefined control codes to 
the simple remote device 1 55 which receives the control 
codes though receiver 421 and passes the information 
to the device control circuitry 425. 
[0042] Finally, the third type of remote device that can 

35 be controlled by the present invention is a conventional 
electronic device that can be controlled by transmitted 
signals such as infrared signals. Therefore, the present 
invention Is backward compatible with most existing de- 
vices that can be controlled remotely. Figure 1e Illus- 

40 trates a typical electronic device 157 that can be con- 
trolled by coded signals. The device has a receiver 450 
for receiving transmitted coded control signals and de- 
vice control circuitry 455 for responding to the coded 
control signals. The hand-held display device 170 con- 

45 trots the electronic device 157, by transmitting coded 
control signals. The remote electronic device 157 re- 
ceives the transmitted signals and responds according 

iy. 

so The Display Device User Interface 

[0043] in order to create a new product that Is easy to 
use, product designers attempt to make the new product 
operate in a manner similar to an old product that the 
55 intended operator already knows how to operate. For 
example, audio cassette tape recorders have been 
available for such a long time that most people know 
how to operate an audio cassette tape recorder. The 
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play, stop, rewind, fast-forward, and record buttons on 
cassette tape recorders are therefore familiar to most 
people. Video cassette recorder manufacturers pro- 
duced easy-to-use video cassette recorders by taking 
advantage of this common knowledge. The video cas- s 
sette recorders are easy to use since the video cassette 
recorders used the same buttons, i.e., the play, stop, re- 
wind, fast-forward, and record buttons, that audio cas- 
sette recorders utilize for operation. 
[0044] To make the hand-held display device 170 w 
easy to use, the present invention displays an intuitive 
graphical user Interface on the hand-held display device 
170. This section defines the graphical user interface 
which is provided to users on the hand-held display de- 
vice 170. 15 
[0045] To design a graphical user interface that is 
easy to navigate, designers should organize the graph- 
ical user interface with a structure that most users al- 
ready understand. Humans live in a three dimensional 
geographic world. To survive in our three dimensional 20 
geographic world people must team how to navigate 
around the local geographic area, including such geo- 
graphic spaces their house, their neighborhood, and 
their town. In order to do this, each person creates a 
mental map of geographic spaces of their relevant local 25 
world. 

[0046] The mental maps of geographic spaces people 
create are de-facto geographic map structures. Figure 
2a illustrates a typical geographic map structure. For ex- 
ample, every person on this planet knows that they live so 
on the Earth. Each person also lives in a certain town; 
yet, each person is aware that other towns exist on the 
Earth. Each person fives at a particular house on a par- 
ticular street; yet, each person is aware that there are 
other streets In the town with other houses on each 35 
street. The identification of one particular house on one 
street within one town on Earth is a naturally occurring 
geographic map structure as illustrated in Figure 2a. 
[0047] The geographic map structure can be extend- 
ed into the house. For example, the living room is one *o 
room of many within the house. The television, televi- 
sion program schedule and the video cassette recorder 
are objects within the living room. Each object can have 
its own geographic space. The map structure can also 
be extended to abstract spaces as wed as geographic 45 
spaces. Each television program listed in a television 
program schedule has a program description. The pro- 
gram description can be considered a space within the 
television program schedule. Most people understand 
both the structure of geographic spaces and the detail so 
that will be found within each geographic space. There- 
fore the de-facto geographic map structure of spaces 
represents a structure that is familiar to most people. 
[0048] The present invention uses the familiar geo- 
graphic map structure of our three dimensional world in » 
a graphical user interface. The present Invention de- 
scribes a technique of user navigation in an arbitrarily 
large and complex geographic map structure through 



graphic background images that depict geographic 
spaces. Such geographic map structures Improve a us- 
er's memory of complex data organization. The advan- 
tage of including such geographic map structures is that 
the familiar structures enable a wider range of people to 
use Information storage facilitated by a computer. 
[0049] Figures 2b and 2c Illustrate a sample geo- 
graphic map structure utilized by the present Invention. 
In the present invention, each Space in the geographic 
map structures of Figures 2b and 2c Is represented as 
a background image on the display screen of hand-held 
display device such as the one illustrated In Figure 1b. 
Located within each geographic Space are objects that 
can be selected and manipulated by the user. 

User Interface Display Elements 

[0050] The user interface environment of the present 
invention consists of two main display elements: "Spac- 
es" and selectable "objects." Each area of the geograph- 
ic map structure of Figures 2b and 2c represents a 
Space. The Space provides a geographic reference that 
informs a user of the user's current location within the 
geographic map structure. Within each Space are one 
or more "objects." A user may select and manipulate the 
objects within each Space. In the present embodiment 
of the hand-held display device 170 with touch screen 
display 37, a user may select objects by touching the 
objects displayed on the touch screen display 37. Point- 
ing devices other than the user's finger and input devic- 
es, other than a touch screen, might handle selection 
and user input in a different manner. 

Spaces 

[0051] Each Space in Figures 2b and 2c has an as- 
sociated background image that contains appropriate 
imagery for the Space. When a user enters a Space, the 
Space's background image is rendered on the display 
screen 37. Each background image contains landmarks 
that help the user identify where the user is. For exam- 
pie, the Earth's surface Space 32 of Figure 2b is illus- 
trated as the background images in Figure 4 and Figure 
5. The Earth's surface background Image of Figures 4 
and 5 resembles a street on the Earth's surface, thereby 
informing a user that the user is at the Earth's surface 
Space of the map structure. The system renders the 
background image using desaturated or dimmed colors 
such that the background image can easily be distin- 
guished from Selectable objects. The background im- 
age for a Space may be larger than the display screen 
37; therefore, the present invention provides a gesture 
for "panning" such that a user may move around within 
a Space as wiN be explained later. 

Objects 

[0052] Objects are individual graphical icons that are 
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rendered within the background images of Spaces. 
Each object is actually an object-oriented programming 
type of object containing data methods which can be In- 
voked to act on the data. A user can select and manip- 
ulate any of the objects which appear within the back- 
ground image of a Space. Each object is brightly colored 
and outlined in black. The bright colors distinguish the 
objects in a Space from the Space's background image 
that is drawn using desatu rated colors. When a user 
points to an object, the user interface generates a "Se- 
lectable Object" sound that informs the user that the ob- 
ject can be selected and manipulated. Referring again 
to Figure 4, in the present example, the houses 47 and 
48 are objects that can be selected and manipulated by 
a user. 

[0053] An "anticipation animation" is used in the 
present invention to inform the user when some action 
is taking place. The anticipation animation is a short an- 
imation displayed on the display screen 37 that is gen- 
erated by the CPU 30. In most cases, the anticipation 
animation consists of enlarging an object's icon such 
that the icon appears to move closer to the user when 
the object is selected. 

[0054] For example, referring to Figure 3, when the 
Earth portal 100 is selected, the Earth portal 100 ex- 
pands such that it appears to move closer to the user. 
After the Earth portal 100 anticipates, the user interfaces 
move the user to the Earth's surface Space 32 that is 
associated with the Earth portal 1 00 as illustrated in Fig- 
ure 4. 

[0055] When a user selects an object by pointing to 
the object for a predetermined minimum period of time, 
a sound is generated, the object anticipates, and the 
CPU invokes a method within the object which performs 
an action associated with the object. 
[0056] There are at least three types of objects that 
exist in the user interface of the present embodiment 
data objects, buttons, and portals. The simplest object 
is a data object. Data objects are objects associated with 
a set of data. Buttons are objects that provide a means 
for accomplishing functions. Portals provide a method 
for moving between different Spaces. Each type of ob- 
ject will be explained individually below. 

Date Objects 

[0057] A data object is a simpie object used to repre- 
sent some piece of information. For example, a data ob- 
ject may represent a color or a television channel. The 
data objects have no methods that can be invoked to 
perform an associated action or function when the data 
object is selected. However, a data object can be 
dropped onto other objects that use the information in 
the data object. For example, a television program list- 
ing data object can be dropped onto a video cassette 
recorder object to program the video cassette recorder 
to record the television program. 
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Buttons 

[0058] A button object Is a special object that has an 
associated action or function. When a user selects a but- 

s ton, the CPU 30 invokes the action method in the button 
object to cause the execution of the action or function 
associated the button. When the button is selected, the 
CPU also anticipates the button and generates a 'button 
action" sound providing feedback to the user that the 

10 button has been selected. Certain buttons provide ad- 
ditional visual feedback to inform the user of the results 
of the method invoked by the CPU to perform the action 
associated with the button. 

[0059] For example, referring to Figure 8, a television 

13 power "on" button 61 and a television power "ofT button 
62 are displayed. When a user selects the television 
power "on" button 61 , the television power "on" button 
61 anticipates and the CPU 30 generates the "button 
action" sound. The CPU 30 also invokes a method In 

20 the button object that transmits the necessary informa- 
tion to turn on a real television set associated with the 
television control panel illustrated in Figure 8. 
[0060] If the real television set is an "intelligent remote 
device", the television set may return a result value to 

25 the display device informing the user whether or not the 
television was turned on. The user Interface may then 
light up the television power "on" button 61 and generate 
a "hum" sound to inform the user that the real television 
was actually turned on. 

90 [0061] Buttons can be copied and moved to different 
Spaces so that a user can customize the "world" of the 
user interface. For example, the television power "on" 
61 and the television power "ofT 62 buttons could be 
copied, and the button copies moved from the television 

33 control Space to any other Space. A user may use the 
copies of the television power "on" 61 and the television 
power "off" 62 buttons in the new Space to control the 
television. 

40 Portals 

[0062] Portals are objects that provide a simple meth- 
od of moving from the current Space to another Space 
associated with the portal. A user moves from the cur- 

43 rent Space to another Space associated with a portal by 
simply Selecting the portal. After a user selects a portal, 
the portal anticipates, generates a "movement through 
portal sound," and the CPU invokes a method that 
moves the user to the Space associated with the portal. 

so [0063] For example, referring to the geographic map 
of Figure 2b, to move from the Earth's surface Space 
32 to the house Space 50, a user simply selects the 
house portal 48. Referring to the Earth's surface Space 
32 as illustrated in Figure 4, a user performs the move 

33 by selecting the house portal 48 within the Earth's sur- 
face Space 32. After being selected, the house portal 
48 anticipates, generates a "movement through portal 
sound," and then the user is moved to the house Space 
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50 as illustrated in Rgure 6. 
[0064] Portals can be duplicated and moved such that 
a user can customize the "world" of the user interface. 
For example, in Rgure 7 a television program schedule 
portal 71 is illustrated within the living room Space 60. 
The TV program schedule porta) 71 is associated with 
a Space that contains TV program schedule informa- 
tion. A user can make a copy of the television program 
schedule portal 71 and move the copy to another Space 
where the user would like to be able to access the tele- 
vision program schedule. The user can then use the 
copy of the television program schedule portal to access 
the same information that could be accessed using the 
original television program schedule portal. Thus, a user 
would be able to access the same information from dif- 
ferent Spaces. 

[0065] There are two special portals that are almost 
always available to an operator of the user interface: a 
Wayback portal and an Agent portal. 

The Wayback Portal 

[0066] The Wayback portal provides a way for a user 
to trace back a path that the user has traveled along 
between different Spaces. The Wayback portal 45 con* 
sists of a small rectangular frame that encloses a snap- 
shot of the previous Space along the path of Spaces 
that the user traveled along. Since the background im- 
age for a Space can be larger than what can be seen in 
a single view, the Wayback portal shows a snapshot of 
the view of the previous Space exactly as It appeared 
when the user went through a portal in that previous 
Space. In the present embodiment, the Wayback portal 
Is available to the user in every Space in the lower right- 
hand comer of the display screen. If desired, the user 
can move the Wayback portal to a new location on the 
display screen such that the Wayback portal will then 
appear at the new location on the screen in every Space. 
[0067] To move one step backward along the current 
path, the user Selects the Wayback portal 45. When Se- 
lected, the CPU 30 anticipates the Wayback portal 45, 
generates a "move through Wayback portal" sound, and 
moves the user to the Space pictured within the Way- 
back portal. In the present embodiment, the "move 
through Wayback portal" sound is actually the same 
sound as the "move through portal" sound except that 
it is played backwards. By playing the "move through 
portal" sound backward when moving through the Way- 
back portal, the movement backward along the current 
path is emphasized to the user. 
[0068] As users go through portals they leave a trail 
of Invisible markers. Each marker represents a visit to 
a Space. The Wayback manages these markers and 
maintains them in the order that the Spaces are visited. 
The markers and the markers 1 order define the path that 
the user has taken through the world of the user inter- 
face. Thus, when a user goes though the Wayback, the 
path defined by the markers is retraced one marker at 



a time. Each time the user goes through the Wayback 
to a Space, a marker is removed from those currently 
managed by the Wayback. 

[0069] For example, referring to geographic map 
5 structure illustrated In Rgure 2b, if a user starts in the . 
outer-space Space 99, moves to the Earth's surface 
Space 32, and then moves to the house Space 50; a 
path win be generated from the outer-space Space 99 
to the Earth's surface Space 32 to the house Space 50. 
io After traversing this path, the display screen 37 wiH dis- 
play the house Space 50 as depicted in Figure 6. In the 
tower right comer of the house Space 50 is the Wayback 
portal 45. As illustrated in Rgure 6, the Wayback portal 
45 displays a miniature snapshot of the previous Space 
13 along the Space path. In this case, the previous Space 
along the Space path is the Earth's surface Space 32; 
therefore, the Wayback portal 45 displays a miniature 
snapshot of the Earth's surface Space 32 as it appeared 
when the user went through the house portal. If the user 
then Selects the Wayback portal 45, the Wayback porta) 
45 anticipates, generates the "move through Wayback 
portal" sound, and then moves the user to the Earth's 
surface Space 32 as depicted In Rgure 4. At the Earth's 
surface Space 32 as depicted in Rgure 4 the Wayback 
now displays a miniature snapshot of the outer-space 
Space 99, which was the first Space along the path. The 
Wayback portal 45 therefore provides a "way back" 
along the path of Spaces the user has traveled. 
[0070] The present invention implements the Way- 
back portal 45 as a stack of data structures, wherein 
each data structure contains a pointer and a pair of co- 
ordinates. The pointer in each data structure on the 
stack points to a Space that the user has previously vis- 
ited. The pair of coordinates In each data structure on 
the stack specifies exactly where the user was within 
the Space pointed to by the pointer when the user left 
the Space through a portal. Each time the user moves 
through a portal from a particular Space, the user inter- 
face "pushes" a data structure, containing the pointer to 
the Space and the coordinate pair which defines exactly 
where the user was within the Space when he went 
through the portal, onto the wayback stack. The Way- 
back portal 45 displays a miniature snapshot of the 
Space that is pointed to by the pointer in the data struc- 
ture on the top of the stack. To trace back along the path 
of Spaces, each time the user Selects the Wayback por- 
tal 45, the user interface "pops 3 a data structure off the 
stack and moves to the Space indicated by the pointer 
In the data structure. 

The Agent Portal 

[0071] The Agent is an anthropomorphic character 
that aids the user in navigating around the user interface 
and In dealing with events that occur In the "world" cre- 
ated by the user interface. Referring to the Earth's sur- 
face Space of Rgure 4, the Agent 41 is illustrated in the 
lower left comer. The Agent 41 carries objects around 
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(or the user. This carrying function is represented by a 
briefcase, which the Agent also carries. A user can ac- 
cess objects that the Agent is carrying by Selecting the 
Agent portal 41 thereby entering an Agent Space asso- 
ciated with the Agent portal 41 . When a user Selects the 5 
Agent porta) 41 . the Agent anticipates by appearing to 
pull out the briefcase and open the briefcase up. The 
CPU then generates the "move through portal" sound 
and moves the user to the Agent Space. 
[0072] To place an object into the Agenfs briefcase, 10 
a user picks up the object and then "Drops" the object 
onto the Agent portal 41 . The object Dropped onto the 
Agent portal 41 is then always accessible to the user 
from any Space by entering the Agenfs Space. The 
Agent is discussed more fully below in the section enti- f 5 
tied "Trie Agent" 

Object Wheel 

[0073] It is often necessary that a user make choices 20 
among an large, arbitrary number of objects. Therefore, 
the user Interface must provide both a method of dis- 
playing a large number of objects and a method of al- 
lowing the user to Select any of the displayed objects. 
[0074] For example, referring to the television control 25 
Space 64 illustrated in Figure 8, the user interface of 
the present Invention must provide a method of display- 
ing an arbitrary number of television channels for the 
user to choose from. In any system, screen space Is at 
a premium, so when the user must choose among many *> 
possible options, the options must be presented com- 
pactly. In the present invention, if there are 500 televi- 
sion channel objects for a user to select from, presenting 
them all at once would not only take up significant 
screen space, but the visual context of the list (i.e., why & 
Is the user looking at this list in the first place) would be 
lost Thus, a method for allowing the user to browse 
through the available objects is required. 
[0075] The user interface of the present invention In- 
troduces an "object wheel" to provide a method for ai- *o 
lowing a user to choose from a large group of objects, 
in the object wheel, a few Immediately accessible ob- 
jects are displayed In an object wheel on the display 
screen while the remaining objects in the group are not 
rendered on the screen. To access the objects not ren- 45 
dcrcd en the screen, the user "Spins' the object wheel 
to move the previously inaccessible objects into view on 
the display screen. 

[0076] Referring to Figure 12, a large number of fruit 
objects are available for a user to choose from, yet only so 
four fruit objects fit horizontally on the display screen 37 
at once. To provide the user with access to all the fruit 
objects available, the present Invention puts the strip of 
fruit objects onto an "object wheel" 38 as displayed in 
the conceptual diagram of Figure 13. A few of the ob- « 
Jects on the object wheel 38 are Immediately accessfcle 
to a user on the display screen 37. To access the objects 
that are not displayed on the display screen 37, the user 



rotates the object wheel 38. As the object wheel 38 ro- 
tates, new objects are brought Into view and other ob- 
jects disappear from view as illustrated in Figure 14. An 
operator uses the navigational gestures Point, Spin , and 
Pickup to manipulate the object wheel presented on the 
touch screen display 37. These three navigational ges- 
tures are explained In the following section. 
[0077] In order to enhance the user's feeling that a 
real wheel is being spun, object wheels obey the laws 
of physics. For example, the object wheels have mass 
and therefore inertia; object wheels spin faster as the 
user applies more "force" when spinning the wheel; ob- 
ject wheels respond to friction since a spinning object 
wheel gradually stops spinning. 
[0078] While an object wheel Is spinning and the ob- 
jects are moving across the screen, the current embod- 
iment of the user interface generates a "clicking" sound. 
The "cUcking" sound generated by the user Interface 
simulates the sound made by detents on a slot machine 
wheel which forces the wheel to stop and be aligned. 
Like a real wheel, the object wheels of the present in- 
vention always stop aligned, and never stop in between 
object choices. A user can stop a spinning object wheel 
at any time by Pointing to a moving object on the object 
wheel while the object wheel is spinning. 
[0079] An object wheel can be aligned vertically or 
horizontally on the display screen. When placed verti- 
cally, the object wheel can be spun both up and down. 
When placed horizontally, the object wheel can be spun 
both left and right 

User Interface Navigational Gestures 

[0080] Navigation and object manipulation in the user 
interface are performed using a screen pointing device. 
In the preferred embodiment, the hand-held display de- 
vice 170 is constructed with a touch screen display 37 
and the pointing device used is a user's finger. Alterna- 
tively, the user interface may be Implemented using any 
other type of screen pointing device such as a mouse, 
a trackball, or a Joystick. This specification , however, will 
only describe the preferred embodiment that uses the 
user's finger as a pointing device on a touch screen. 
[0081] in the preferred embodiment of the present In- 
vention, the hand-held display device 170 comprises, In 
part, a touch screen display 37 as illustrated in Figure 
1b. The touch screen display 37 senses when a finger 
has been placed on the display screen and identifies 
where the finger has been placed. Users select objects 
on the touch screen and manipulate the objects using a 
set of six navigational gestures: Point, Select, Pan, Spin, 
Pickup, and Drop. Each navigational gesture used by 
the user interface is explained Individually below. (Since 
the navigational gesture names are the same as com- 
mon words, this specification will capitalize aH referenc- 
es to the navigational gestures.) 
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Point 

[0082] Point Is the most simple gesture in the user In- 
terface of the present invention. An operator uses the 
Point gesture to pick objects rendered on the display 
screen. For example, if the user Interface of the present 
invention requests the user to choose one of many ob- 
jects rendered on the display screen, the user Points to 
an object to choose the object. 
[0083] A user Points to an object by simply placing a 
finger on the object displayed on the touch screen. 
When a user has successfully Pointed to an object, the 
user interface generates a "Selectable Object" sound. 
The Selectable object sound provides feedback to the 
user Indicating that the user has successfully Pointed to 
the object. 

Select 

[0084] The Select gesture is very similar to the Point 
gesture. An operator uses the Select gesture to have an 
object invoke a method to perform an function or action 
associated with the object. For example, a Selected por- 
tal Invokes a method that moves the user to the Space 
associated with the portal and a Selected button invokes 
a method that performs the action or function associated 
with the button. 

[0085] To Select an object, a user first Points to an 
object by touching the object with a finger. If the user 
continues to Point to the object for a predetermined min- 
imum period of time, the object becomes Selected. After 
a user Selects an object, the object "anticipates," gen- 
erates a sound for user feedback, and then a method 
within the object Is invoked. If the object is a button, then 
the method invoked performs a function associated with 
the button. If the object is a portal, then the method in- 
voked will move the user to the Space associated with 
that portal. 

[0086] Alternatively, a user can Select an object on 
the display screen by quickly Pointing to the object twice 
in succession. When a user Selects an object by quickly 
Pointing to the object twice, the object does not antici- 
pate. Instead, the user interface only generates the ap- 
propriate feedback sound and then performs the action 
associated with the object This provides the user a 
method of Selecting an object quickly. 

Pan 

[0087] To move around within a Space a user uses 
the Pan navigational gesture. A user Pans though a 
Space by placing a finger on the background image of 
a Space displayed on the touch display screen 37 and 
sliding the finger in the desired direction of movement 
The user interface of the present invention responds by 
generating a "Pan Active" sound and moving back- 
ground image of the Space and all the objects within the 
Space In the opposite direction of the finger's move- 



ment. By moving the Space in the opposite direction, 
the user Interface simulates movement of the users* 
point of view In the other direction. 
(0088] Figure* 4 and 5 illustrate the use of the Pan 
5 gesture. If a user places his finger on the background of 
Figure 4 and then slides the finger on the touch screen 
to the right, the view will correspondingly pan to the right 
by moving the background Image of the Space and all 
the objects within the Space to the left. Figure 5 Ulus- 
10 trates the result of the Pan right operation where house 
48 has moved from the right side of the display screen 
over to the left side of the display screen. 
[0089] The background image for each Space in the 
present invention can be arbitrarily large. If the back- 
's ground Image of a Space has finite size, the user Inter- 
face of the present invention provides feedback that the 
furthest extent of the background image has been 
reached. When a user Pans in a direction all the way to 
the end of a background Image, the Pan gesture will not 
20 move the user any further In that direction. To Inform the 
user of this fact, the user interface generates an "end of 
extent" sound instead of the "Pan Active" sound when 
a user attempts to Pan past the end of a background 
image. The "end of extern" sound sounds like a "thud" 
23 made when something bumps into a wall. The "end of 
extent" sound thereby informs the user that the user has 
hit the end of the background Image and can go no fur- 
ther. 

so Spin 

[0090] An operator uses the Spin gesture to browse 
through several objects placed on an object wheel. 
When a user Spins an object wheel, the visible objects 
$5 on the object wheel move across the display screen and 
disappear off one end of the screen and previously in- 
visible objects move onto the display screen at the other 
end. 

[0091] A user Spins an object wheel by placing a fin- 
40 ger on the object wheel and then moving the finger 
across the display screen In a direction parallel to the 
object wheel's movement. Figure 1 5 illustrates the Spin 
gesture. In Figure 1 5, a user first places a finger on the 
object wheel (as shown by the asterisk 96). The user 
45 then slides the finger across the object wheel (as shown 
by the arrow 97). This finger gesture causes the object 
wheel to Spin in the direction of the fingers movement 
The faster the user slides the finger across the object 
wheel on the display, the faster the object wheel spins. 
so Once the object wheel is spinning, the objects on the 
object wheel appear at one end of an object wheel, 
move across the display screen, and then disappear off 
the other end of the object wheel similar to figures which 
appear on a slot machine wheel. 

55 

Pickup 

[0092] The Pickup gesture is used to pickup an object 
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located within a Space or on an object wheel such that 
a user can cany the object around the "world" defined 
by the user interface of the present invention. In the 
present embodiment, different methods are used to 
Pickup an object from a Space or an object wheel. Each 
method will be explained Individually below. However, it 
will be appreciated that the same gesture could be used 
to pickup objects In both situations. 
[0093] The gesture used to Pickup an object from an 
object wheel is illustrated in Figure 16a. Figure 16a Il- 
lustrates an object wheel 91 with a number of fruit ob- 
jects to choose from. To Pickup one of the fruit objects 
disposed on the object wheel 91, a user first places a 
finger on the object (as designated by the asterisk 92) 
and moves the finger In a direction perpendicular to the 
motion of the object wheel. The movement of the finger 
in a direction perpendicular to the motion of the object 
wheel is illustrated by arrow 93. The perpendicular fin- 
ger motion "tears" the object off the object wheel such 
that the user can move the object. When the object 
comes loose from the object wheel, the user interface 
generates a "tearing" sound that informs the user that 
the object has been removed from the object wheel. 
When an object has been picked up, the object casts a 
shadow onto the background image of the Space and 
moves the object along with the user's finger. 
[0094] The gesture used to Pickup an object from a 
Space is illustrated in Figure 16b. Figure 16b illustrates 
a Space with an object In the center. To Pickup the ob- 
ject, a user circles the object with the finger as illustrated 
by arrow 203 and then touches the object as Illustrated 
by asterisk 201 . When the object comes loose from the 
Space, the user interface generates the tearing sound 
that informs the user that the object has been picked up. 
After an object has been picked up from a Space by a 
user, a ghost image of the object remains at the object's 
original location. The ghost image of the object is drawn 
in desaturated colors indicating to the user that the ghost 
object cannot be Selected. 

[0095] After a user has picked up an object, either 
from an object wheel or a Space, the user Interface gen- 
erates a "carrying object" sound that informs the user 
that an object is being carried. The user can move the 
carried object within a Space, from Space to Space, or 
Drop the carried object onto another object. 

Drop 

[0096] The final gesture used to Interact with the user 
interface of the present invention is Drop. After a user 
has used the Pickup gesture to pick up an object from 
a Space or from an object wheel, the user must continue 
to keep the finger pressed against the screen to carry 
the object. The user moves the carried object by sliding 
the finger across the touch screen display system. After 
moving the carried object to a desired location, the user 
then uses the Drop gesture to place the object at the 
desired location. To drop the object the user lifts the fin- 



ger off the touch screen display. When the user Drops 
an object, the object fails Into the background image of 
a Space as if pulled by gravity, and the object no longer 
casts a shadow. 

5 [0097] A user of the present Invention uses the Drop 
gesture to perform a number of different functions. Trie 
Drop gesture can be used to copy an object by picking 
up the object and Dropping the object into a portal such 
that a copy of the object is created In the Space asso- 

io dated with the portal. The Drop gesture can be used to 
move an object to a new location by picking up the object 
and Dropping the object at the desired location. The 
Drop gesture can also be used to provide a data object 
containing information to another object which uses the 

15 Information to perform a function. Each different use of 
the Drop gesture wiH be explained below. 
[0098] To make a copy of an object using the Drop 
gesture, a user must first Pickup the object and then 
Drop the object onto a portal. An object that has been 

20 picked up is carried over the portal by sliding the finger 
across the screen. Once the carried object is over the 
portal, the portal becomes highlighted such that It pro- 
vides feedback to the user signaling that the carried ob- 
ject is over the portal. In the present embodiment, the 

25 portal emits a green glow. To Drop the carried object Into 
the portal, the user lifts the finger away from the touch 
screen display. The object Is then displayed such that it 
passes through the portal with an appropriate anima- 
tion. During the animation, the CPU 30 generates a 

90 "sucked through portal" sound effect. When the object 
has completely disappeared through the porta), the 
ghost image of the object drawn in desaturated colors 
in the present Space changes back into a brightly color- 
ed object. This provides visual feedback to the user that 

35 the original object remains in the present Space even 
though a copy of the object has been sent through the 
portal to another Space. 

[0099] Users can move objects either within a single 
Space or from one Space to another. To move an object 

40 within a single Space a user first uses the Pickup ges- 
ture to pick up an object, moves the object to the desired 
new location, and then Drops the object using the Drop 
gesture. After the object Is Dropped, the ghost image of 
the object merges with the Dropped object at the new 

45 location to form a single brightly colored image of the 
object at the new location. This informs the userthatthe 
object has been moved to a new location and thai a copy 
has not been made. 

[01 00] Moving an object to another Space Is accom- 
50 plishedbycarryingtheobjectmroughaportaitoanother 
Space. To move an object to another Space, a user must 
first Pickup the object and then carry the object over a 
portal. Once the carried object is over the portal, the por- 
tal becomes highlighted such that It provides feedback 
55 to the user signaling that the carried object is over the 
portal, and the portal emits a green glow. To move the 
carried object through the portal, the user holds the ob- 
ject over the portal for a predetermined minimum 
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amount of time. The portal eventually anticipates mov- 
ing the user and the carried object to the Space associ- 
ated with the portal. The ghost image of the carried ob- 
ject drawn In desatu rated colors is also moved along, 
with the user, and the carried object to the destination 5 
Space. When the user Drops the carried object by lifting 
his ringer off of the touch screen display system, the car- 
ried object and the ghost image merge into a single 
brightly colored image of the object This provides feed- 
back to the user that the object has been moved to the *o 
Space and has not been copied. 
[01 01 ] The Drop gesture can also be used to provide 
Information to an object A user provides information to 
an object by Dropping an object containing data (i.e. a 
data object) onto the object which will use that data. 15 
[0102] To provide information to an object, a user 
must first Pickup a data object that contains the neces- 
sary information. The user then carries the data object 
over the object which can utilize the information. When 
the carried data object is over the object that can utilize 20 
the information, the object that can utilize the informa- 
tion becomes highlighted if the carried data object Is of 
the correct class. By becoming highlighted, the object 
that can utilize the information provides feedback to the 
user that the carried data object is of the correct class. & 
To Drop the data object into the object that can utilize 
the information, the user lifts his finger used to carry the 
data object away from the touch screen. The data object 
is then passed through and Into the portal with appro- 
priate animation and sound effects. When the data ob- &> 
ject has completely disappeared Into the receiving ob- 
ject which can use the information, the receiving object 
then performs the appropriate function on the data pro- 
vided. 

[0103] For example, the Drop gesture can be used to 35 
provide television program information to the video cas- 
sette recorder object which Is associated with a physical 
video cassette recorder, such that the video cassette re- 
corder programs itself to record the television program. 
To program the video cassette recorder, a user first uses 4> 
the Pickup gesture to pickup a data object containing 
television program Information. The user then carries a 
television program data object over the video cassette 
recorder object The video cassette recorder object then 
modifies its appearance to indicate it is a "program video 4S 
cassette recorder mode. This informs the user that the 
video cassette recorder "understands" the information 
that Is over It (i.e., the video cassette recorder will be 
programmed to record the television program if the user 
Drops the television program data object onto the video so 
cassette recorder). 

[0104] When the user Drops the television program 
object Into the video cassette recorder, the video cas- 
sette recorder object recognizes that the television pro- 
gram data object contains Information it can use. The 55 
video cassette recorder then extracts the time and chan- 
nel information from the television program data object 
such that the video cassette recorder object programs 



itself to record the television program. 

A Tour of the Present Embodiment In Operation 

[0105] To best convey the look and feel of the user 
interface of the present Invention, a "tour" through a 
sample environment is provided with reference to Fig- 
ures 3-10. Hie sample environment explored In the 
tour is based upon the geographic map structure of 
Spaces diagrammed in Figures 2b and 2c. The tour 
through the sample environment described herein is not 
intended to limit the scope of the present Invention, but 
instead to provide an example of one possible embodi- 
ment of the present invention. Due to the limitations of 
a written Specification in describing the present Inven- 
tion's animated user interface, the reader Is encouraged 
to view a video tape entitled "Green Project User Inter- 
face," submitted by the applicant concurrently with the 
filing of the application on which this patent is based. 
[0106] Referring to Figure 3, upon first entering the 
user interface of the present invention, the user interface 
initially positions the user in an outer-space Space 99. 
This Space is illustrated in the geographic map structure 
of Spaces diagrammed in Figure 2b. Although, the user 
is initially placed in the outer-space Space 99, the user 
interface will later start the user at the last Space the 
user was In when the user quit using the hand-held dis- 
play device. Referring again to Figure 3, at the outer- 
space Space 99, the user interface renders a brightly 
colored Earth portal 100 against a dark outer-space 
background Image on the display screen. 
[0107] Soon after entering the user interface, an out- 
line of a hand 31 appears on the display screen to teach 
the user how to navigate around the user Interface. The 
hand 31 that appears on the display screen Is the hand 
of the Agent. The Agenfs hand 31 extends a finger to 
touch the screen and then slides the finger across the 
screen causing the Space and objects on the screen dis- 
play to Pan and the user interface to emit the "Panning 
active" sound. Referring to Figure 3, the outline hand 
31 of the Agent Is shown moving across the screen. By 
touching the display screen and moving a finger across 
the display screen, the Agenfs hand 31 demonstrates 
to a new user how to use the Pan gesture to move 
around the view of a Space. 
[0108] After demonstrating how to use the Pan ges- 
ture to move around in a Space, the hand 31 of the Agent 
next demonstrates how to use the Point and Select ges- 
tures. The Agent demonstrates how to use the Point 
gesture to Point to an object by placing a finger on the 
Earth portal 1 00. After the Agent Points to the Earth por- 
tal 100, the user interface emits the "Selectable Object" 
sound to provide feedback which informs the user that 
the Earth portal 100 is a Selectable Object The Agent 
then demonstrates how to use the Select gesture to Se- 
lect an object by continuing to Point to the Earth portal 
100. After the Agent's hand 31 Points to the Earth portal 
1 00 for the predetermined minimum amount of time, the 
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Earth porta! 1 00 anticipates and the CPU generates the 
"move through portaP sound. The Earth portal 100 an- 
ticipates by expanding rapidly such that the Earth portal 
100 appears to move closer to the user. The screen then 
fades and a new background image appears displaying 5 
the Earth's surface Space 32 associated with the Earth 
portal 100, as illustrated In Figure 4. 
[0109] At the Earth's surface Space 32, the Agent por- 
tal 41 appears to "jump" onto the touch screen display 
37 and moves to the lower left comer of the display to 
screen. The Agent 41 wil always be accessible to the 
user because the agent appears In every Space, except 
the Agent's Space. If desired, the user can move the 
Agent from the lower left comer to a different location 
on the display screen. This movement of the Agent wiH 
change the Agent's location in every Space. 
[01 10] Referring to Figure 4, In the lower right comer 
of the display is the Wayback portal 45. The Wayback 
portal 45 displays a framed snapshot of the outer-space 
Space 99 from Figure 3 which the user just came from. 20 
If the user Selects the Wayback portal 45, the user in- 
terface of the present invention moves the user back 
along the traveled path to the Space illustrated within 
the Wayback portal 45. For example, If the user desires 
to return to the outer-space Space 99 displayed in the & 
Wayback portal 45, the user simply Selects the Way- 
back portal 45. 

[01 1 1 ] The Earth's surface Space 32 depicted in Fig- 
ure 4 is comprised of a number of houses and commu- 
nity buildings such as stores, banks, schools, etc. Each *> 
building In the Earth's surface Space 32 represents a 
portal that can be Selected by the user to move the user 
to a Space associated with the portal. For example, in 
the Earth's surface Space depicted In Figure 4, the first 
house 47 and the second house 48 are portals which & 
can be Selected by the user. If the user Selects either 
the first house 47 or the second house 48, the user will 
be moved to an associated house Space. 
[01 12] While within the Earth's surface Space depict- 
ed in Figure 4, the user can move around in the Space *o 
by using the Pan gesture on the touch screen as dem- 
onstrated by the Agenfs hand 31 In Figure 3. To Pan, 
the user simply places a finger on the screen and moves 
the finger in the direction the user wishes to move. The 
user interface responds to the user's finger motion by 
generating the "Panning Active" sound and moving the 
background image and the objects on the screen in the 
opposite direction of the finger's movement, thereby 
bringing other objects into the view. 
[01 1 3] For example, If a user viewing the Earth's sur- so 
face Space 32 as depicted in Figure 4 wishes to move 
to the right, the user simply places a finger onto a back- 
ground area of the Earth's surface Space 32 and slides 
the finger to the right. The user interface reciprocally re- 
sponds to the user's finger motion by moving the 55 
Space's background image and the objects In the Space 
to the left. Figure 5 depicts the view of the Earth's sur- 
face Space 32 after the user has placed a finger onto a 



background area of Figure 4 and slid the finger to the 
right A first house portal 47 displayed In Figure 4 has 
moved out of the view and Is not visible In Figure 5. A 
second house portal 48 displayed on the right side of 
the view in Figure 4 has moved to the left side of the 
view in Figure 5 and now is only partially visfcle. A bank 
portal 49 that was not visible in Figure 4 has moved Into 
view from the right as illustrated In Figure 5. The Agent 
41 and Wayback portal 45 remain in the view in the same 
lower left and lower right positions respectively. 
[0114] When the user Selects any portal from the 
Earth's surface Space 32 as depicted in Figures 4 and 
5, the Selected portal anticipates to provide feedback to 
the user and the user interface moves the user to a 
Space associated with the Selected portal. For exam- 
ple, when a user at the Earth's surface Space 32 depict- 
ed in Figure 4 Selects the first house portal 47, the 
house portal 47 anticipates and then the user interface 
moves the user to a house Space associated with the 
house portal 47. Figure 6 illustrates a house Space 50 
associated with the house portal 47 of Figure 4. 
[0115] The house Space 50 displayed in Figure 6 
consists of a house with the outer walls removed such 
that the individual rooms inside the house are visible. 
All of the rooms inside the house, such as the Irving room 
54, bedroom 53, bathroom 55, attic 52, and kitchen 56 
depicted In Figure 6, are portals that can be Selected 
by the user. As illustrated in Figure 6, the Wayback por- 
tal 45 contains a miniature snapshot of the Earth's sur- 
face Space 32, as it appeared when the user left the 
previous Space. Thus the Wayback portal 45 depicted 
in Figure 6 not only contains a miniature view of the 
Earth's surface Space, but a view of that portion of the 
Earth's surface Space 32 that was in view when the user 
Selected the house portal 47 of Figure 4. To move back 
up to the Earth's surface Space of Figure 4, the user 
Selects the Wayback portal 45 that contains a snapshot 
Image of the Earth's surface Space. 
[0116] Referring to Figure 6, if the user Selects the 
living room portal 54, the living room portal 54 antici- 
pates, the "move through portal" sound is generated, 
and the user Interface moves the user to a living room 
Space associated with the Irving room portal 54. Refer- 
ring to Figure 7, an embodiment of a living room Space 
60 Is depicted. Within the living room Space 60 of Figure 
7 is a television 73, a video cassette recorder 75, and a 
television program schedule book 71 . The television 73, 
video cassette recorder 75, and television program 
schedule book 71 are each portals that can be Selected 
by the user. The television 73 and the video cassette 
recorder 75 portals are associated with a real television 
and video cassette recorder respectively. 
[0117] When a user Selects the television portal 73 
displayed within the living room Space 60 of Figure 7, 
the television portal 73 anticipates and the user Interface 
displays a graphical user interface associated with the 
real television. Figure 8 depicts the television control 
Space 64 which comprises the television's graphical us- 
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er interface in the present embodiment. The objects 
available In the television Space 64 Include a green "on" 
button 61, a red "off* button 62, and several television 
channel buttons 66 - 69. The television channel buttons 
66 - 69 are on a horizontally aligned object wheel 63 
such that a user can Spin the object wheel 63 left or right 
in order to reveal additional television channel buttons 
that are not currently displayed on the display screen. 
[01 1 6] The on button 61 , off button 62, and television 
channel button 66 - 69 objects available in the television 
Space 64 are not portals that give access to other Spac- 
es in the Space map of Figure 1 b. Instead, the on button 
61 , off button 62, and television channel buttons 66 - 69 
are buttons that are associated with functions. For ex- 
ample, the green "on" button 61 and red "ofT button 62 
are associated with the functions "turn the television on" 
and turn the television off" respectively. When a user 
Selects either the on or off button, the button anticipates 
to provide feedback to the user that the button was Se- 
lected. The hand-held display device then transmits the 
appropriate commands that turn the television on or off. 
Similarly, if the user Selects any of the television channel 
buttons 66 - 69, the television channel button anticipates 
and the hand-held display device transmits the appro- 
priate commands to the real television thereby tuning 
the television to the Selected channel. The process of 
transmitting commands to the physical television set Is 
explained in detail in the following section. 
[0119] Referring back to Figure 7, if a user Selects 
the television program schedule book portal 71 , the tel- 
evision program schedule book portal 71 anticipates, 
and the user interface moves the user to the television 
program schedule Space. Figure 9 depicts the televi- 
sion program schedule book Space of the present em- 
bodiment In the television program schedule book 
Space of Figure 9, the user is presented with a two di- 
mensional array of television program listings. The array 
organizes the television program listings vertically by 
television channel and horizontally by time. Each tele- 
vision program listing is a portal that can be Selected by 
the user. Each television program listing can also be car- 
ried by the user and Dropped onto the video cassette 
recorder portal such that the hand-held display device 
programs the video cassette recorder to record the tel- 
evision program. 

[01 20} The television program schedule book Space 
of Figure 9 Is actually organized as a pair of connected 
vertically oriented object wheels. Referring to Figure 1 1 , 
a conceptual diagram of the television program sched- 
ule book Space is illustrated. The television channels of 
Figure 9 are placed on a first object wheel 72 as illus- 
trated in Figure 11. The television program listings are 
placed a second object wheel 65. The two object wheels 
72 and 85 are connected such that if the user Spins ei- 
ther object wheel, the other object wheel wiD spin at the 
same rate such that the television channels and televi- 
sion program listings are always aligned. 
[0121] Each television program listing on the televi- 



sion program schedule object wheel 85 Is a portal. When 
a user Selects one of the television program listing por- 
tals, the television program listing portal anticipates and 
the user interface moves the user to a television pro- 

5 gram Information Space. Figure 10 Illustrates a typical 
television program Information Space. The television 
program information Space Includes a dock 81 , a date 
82 and television channel 83 which Inform the user when 
the television program Is on and what channel the tele- 

10 vision program Is on. The television program Information 
Space also includes a description of the television pro- 
gram as illustrated in Figure 10. 
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[0122] The Agent is an anthropomorphic character 
designed to help a user of the present Invention In a 
number of ways. The Agent assists users by providing 
help, providing state Information, alerting the user to cer- 
20 tain events, and acting as a repository. Each function 
performed by the Agent will be explained individually. 

Help 

25 [0123] As previously described in the tour of the 
present embodiment, with reference to Figure 3, when 
a user first enters the user interface of the present In- 
vention, the Agent demonstrates how to use the Point 
gesture and the Select gesture. Furthermore, the Agent 

so demonstrates how to use the Pan gesture to move 
around within a Space. The Agent can also demonstrate 
how to use other features of the user interface as nec- 
essary. 

[0124] To further aid users, the Agent carries a help 
35 manual in its briefcase that is available to the user at 
any time. To access the help manual, the user simply 
Selects the Agent portal. The Agent anticipates by pro- 
viding an animation of the Agent opening up Its brief- 
case. The user then Selects the help manual from within 
40 the Agent's briefcase. 

State and Mode Information 

fp1 25] The Agent is also used to express the current 
45 state or mode of the user interface. To express the cur- 
rent state or mode the Agent changes its clothing, 
changes Its expression, or changes the props it Is car- 
rying. 

[0126] For example, if the user interface of the present 
so invention is placed into a diagnostic mode, the clothing 
that the Agent Is wearing Is changed to convey this fact. 
Referring to Figure 1 7a, the Agent is shown in a doctor's 
uniform to symbolize that the user Interface is In a diag- 
nostic mode. 

55 [0127] Figure 17b provides another example of how 
the Agents appearance can convey state Information. 
When the user is in a writing state, the Agent Is given a 
pencil to symbolize that the user interface is in a writing 



16 



EP0 626 635B1 



32 



31 

state. 
Alerts 

[01 28] While a user is using the hand-held display de- 
vice of the present invention, certain events may occur 
that require the user to be Informed immediately. Such 
events are usually referred to as alerts. For example, if 
the hand-held display device is coupled to an electronic 
messaging service, the hand-held display device should 
alert the user when a message has been received that 
is addressed to the user. 

[01 29] In the user interface of the present invention, 
the Agent is used to obtain the user's attention In order 
to inform the user about any alerts. To get the user's 
attention the Agent waves its hand. Additionally, if the 
alert is very Important, the Agent may wave both arms 
or produce sounds to further attract the user's attention. 
[0130] For example, when an event occurs that re- 
quires the user's attention, the Agent begins waving as 
illustrated in Figure 17c. To explain what caused the 
alert, the Agent's appearance may be further altered as 
demonstrated in the previous section in order to convey 
Information about the specific alert For example, when 
an urgent message arrives, the Agent can wave with 
one hand while holding a telegram in the other hand. 
[01 31 ] Since alerts are generally more important than 
state or mode information, alerts will take precedence 
over state or mode Information. Therefore, if the image 
of the Agent is currently being used to convey state or 
mode information when an Important event occurs, the 
image of the Agent will change to get the user's attention 
and inform the user about the alert For example, if the 
Agent is holding a pencil while In writing mode, as illus- 
trated in Figure 17b, when an urgent message arrives, 
the Agent will immediately change to the waving anima- 
tion as illustrated in Figure 17c to inform the user about 
the message. 

Remote Device Control 

[01 32] The hand-held display device of the present in- 
vention is designed to control remote devices. Accord- 
ingly, the user interface defined In the previous section 
for the hand-held display device is designed to be an 
Ideal graphical user Interface for controlling remote de- 
vices. For example, to control a device in the user's liv- 
ing room, a user first moves to the living room Space 60 
of the user Interface as illustrated in Figure 7. The user 
then Selects an object corresponding to a device that 
the user wishes to control. For example, If the user de- 
sires to control a television, the user Selects the televi- 
sion portal 73 associated with a real television. When 
the user Selects the television portal 73, the television 
portal 73 anticipates and then displays a graphical user 
Interface for controlling the physical television in the us- 
er's Irving room. Figure 8 illustrates one possible em- 
bodiment of a set of television controls for controlling a 



television. The user then controls the television using 
the set of television controls displayed on the hand-held 
display device. 

[01 33] In order for a remote device to be controlled by 

5 the hand-held display device, the hand-held display do- . 
vice must have certain Information about the remote de- 
vice. Specifically, the hand-held display device must be 
provided with an object-oriented programming type of 
object that is associated with the remote device. The ob- 

10 Jed comprises a plurality of methods including a method 
for displaying the object on the display screen of the 
hand-held display device and a method for displaying a 
graphical user interface for the remote device. 
[0134] To simplify the operation of the present inven- 

15 tion, each compatible remote device automatically 
transmits the required object-oriented programming 
type of object to the hand-held display device. When a 
remote device transmits the object to the hand-held dis- 
play device, the hand-held display device invokes the 

20 method for displaying the object on the display screen. 
The user can then move the object to an appropriate 
Space. If the user Selects the object, the Selected object 
anticipates and then the method for displaying the 
graphical user interface is invoked. The method displays 

25 the remote device's graphical user interface on the 
hand-held display device's touch screen display. The 
user can then control the remote device by Interacting 
with the remote device's graphical user interface dis- 
played on the touch screen display. 

30 [0135] The hand-held display device of the present in- 
vention automatically obtains a graphical user Interface 
from two types of compatble remote devices. The first 
type of compatible remote device is the intelligent re- 
mote device with a dedicated computer control system. 

6 The second type of compatible remote device is a sim- 
ple remote device that can only respond to simple coded 
commands. 

[0136] The hand-held display device Interacts differ- 
ently with the two types of compatible remote devices. 

40 The hand-held display device communicatee with intel- 
ligent remote devices using peer-to-peer communica- 
tion with remote method Invocation for remote device 
control. The hand-held display device communicates 
with simple remote devices using only simple control 

45 codes. However, from the user's perspective the differ- 
ences between the intelligent remote devices and the 
simple remote devices cannot be seer., and the user's 
interaction with the graphical user interface is identical 
for both types of remote devices. The implementation 

so detail of whether a user's actions cause a remote meth- 
od invocation, or transmission of control codes through 
some communication mechanism is hidden from the us- 
er. 

[01 37] This section will explain how the two types of 
55 compatible remote devices transmit information to the 
hand-held display device. Furthermore, a third type of 
remote device win explained. The third type of device 
consists of any electrical device that can be controlled 
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by coded infrared signals. 

[0138] Within this section, reference will be made to 
object-oriented programming type objects. Each object- 
oriented programming type of object contains data and 
methods that can be Invoked to perform functions. To 
simplify the terminology used, the term "program object" 
will be used when referring to an object-oriented pro- 
gramming type of object. To eliminate confusion, the ob- 
jects displayed on the touch screen display in the user 
interface disclosed in the previous user interface will be 
referred to as "display objects" In this section. 

Peer to Peer Communication With Remote Method 
Invocation 

[0139] In the present invention, a remote device is de- 
fined as an "Intelligent remote device" if that remote de- 
vice contains a microprocessor, a communication sys- 
tem, and a sufficient amount of memory such that it can 
invoke methods in a program object. Figure 1c illus- 
trates, in block diagram form, the components in an in- 
telligent remote device. The hand-held display device of 
the present invention communicates with Intelligent re- 
mote devices in a peer-to-peer fashion. 
[01 40] The intelligent remote device has an associat- 
ed graphical user interface comprising touch screen 
control elements such as dials, switches, sliders, etc. 
The intelligent remote device's graphical user interface 
is defined within a user interface program object 303 
stored In the intelligent remote device's memory 404. 
When an intelligent remote device is first turned on, the 
intelligent remote device automatically transmits its user 
interface program object 303 to the hand-held display 
device. The hand-held display device can then display 
the user interface for the intelligent remote device as 
defined by the user Interface program object. 
[0141] When a user interacts with the intelligent re- 
mote device's graphical user interface as displayed on 
the hand-held display device, the hand-held display de- 
vice invokes methods in a device driver program object 
301 within the intelligent remote to control the remote 
device. Thus the software that actually "controls" an In- 
telligent remote device Is located within the intelligent 
remote device Itself. 

[0142] The procedure of transmitting the user inter- 
face object 303 from an intelligent remote device to the 
hand-held display device will be disclosed with refer- 
ence to Figures 18a - 18d, 19a - 19d and the flowchart 
in Figure 20. Figures 18a - 1 8d illustrate, on a high lev- 
el, how the intelligent remote device 160 and the hand- 
held display device 170 of the present Invention com- 
municate with each other. Figures 19a -19d illustrate, 
in conceptual form, how the user Interface program ob- 
ject 303 in the intelligent remote device 150 Is transmit- 
ted to the hand-held display device 170. 
[01 43] Referring to step 1 01 of Figure 20, when a new 
intelligent remote device 150, such as the intelligent 
thermostat control device in Figure 16a, Is first turned 



on, the new Intelligent remote device 150 broadcasts 
across the communications network 160 that it has a 
user interface program object to export. As set forth in 
the hardware section, the communication on the com- 

5 munications network 160 can take place using radio 
waves, low power, cellular, Infrared signals, or electrical 
signals on existing power Ines In the case of an Eche- 
lon® base LON© system. In step 102, a hand-held dis- 
play device 170 receives the interface export message. 

io The hand-held cteplay device 170 signals back across 
the communications network 160 to the new intelligent 
remote device 150 requesting the new Intellgent device 
150 to begin transmitting the user Interface object 303. 
[0144] The Intellgent remote device 150 then trans- 

15 mits the user Interface object 303 to the display device 
170 as specified In step 103 of Figure 20 and illustrated 
in Figure 18a. Referring to Figure 19a, the Intelligent 
remote device 1 50 also transmits a pointer (as indicated 
by dotted lines) to a device driver program object 301 

20 within the intelligent remote device 1 50. The user Inter- 
face program object 303 defines how the user interface 
for the intelligent remote device 150 should appear on 
a display screen. The user Interface program object 303 
Invokes methods in a device driver program object 301 

25 to handle user interface interactions. The device driver 
program object 301 contains methods that control the 
actual Intelligent remote device 150. Referring to Figure 
19b, the display device 170 now has a copy of the intel- 
ligent device's user interface program object 313 and a 

so device driver program object 31 1 pointer. 

[01 45] After receiving the user interface program ob- 
ject 313, the hand-held display device 170 invokes a 
method in the user interface program object 313 that 
renders a display object In the current Space as stated 

35 in step 104. The new intelligent device 150 also registers 
with a remote name server connected to the communi- 
cations network 160. 

[0146] When a user Selects the display object asso- 
ciated with the new intelligent remote device 150, the 

40 hand-held display device 170 invokes a method In the 
user interface program object 313 as stated in step 105 
(the object with an invoked method is shown in bold). 
The method invoked in the user interface program ob- 
ject 313 displays the remote intelligent device's graphi- 

45 cal user interface on the display screen 171 of the dis- 
play device 1 70 as illustrated In Figure 18b. A user can 
Interact with the remote intelligent device's graphical us- 
er Interface displayed on the display device 1 70 as stat- 
ed in step 106. 

50 [0 1 47] The user controls the intelligent remote device 
150 by Interacting with the graphical user Interface dis- 
played on the hand-held display device 170. To resolve 
the user's Interactions, the user Interface program ob- 
ject 313 requires control methods in the Intelligent re- 

55 mote device's device driver program object 301 . To ac- 
cess the control methods In the intelligent remote de- 
vice's device driver program object 301 , the hand-held 
display device 170 uses the device driver program ob- 
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ject pointer 31 1 . The device driver program object point- 
er 311 Invokes the methods In the real device driver pro- 
gram object 301 . The Intelligent remote device's device 
driver program object 301 b Invoked by sending a mes- 
sage over the communication network 160 as illustrated 
in Figures 18c and 19c (again, the object with an in- 
voked method is shown in bold). The methods invoked 
in the Intelligent remote device's device driver program 
object 301 control the intelligent device 1 50 as stated in 
step 108. 

[01 48] Finally, in step 1 09, any return values or feed- 
back generated by the device driver program object 301 
in intelligent remote device 150 are sent back to the 
hand-held display device 1 70 over the communications 
network 160 as illustrated in Figures 18d and 19d. The 
user interface program object 313 changes the graphi- 
cal user Interface on the display screen 1 71 In response 
to the return values or feedback received from device 
driver program object 301 in the intelligent remote de- 
vice 150. 

[0149] A robust communication protocol across the 
communications network 160 is required to communi- 
cate remote method invocations between the hand-held 
display device 170 and the intelligent remote device 
150. Both the hand-held display device 170 requesting 
the method Invocation and the intelligent remote device 
150 that executes the method must be able to handle a 
broken communication path. Therefore, both the hand- 
held display device 170 and the intelligent remote de- 
vice 150 have a dedicated remote procedure call (RPC) 
manager that handles the remote method Invocation. 
Referring to Figures 1 9a - 19d, the remote procedure 
call manager 309 transmits the remote method invoca- 
tion request for the hand-held display device 170. On 
the other side, a server remote procedure call manager 
307 processes the received remote method invocation 
in an intelligent remote device 150. 
[01 50] Figures 21 a and 21 b illustrate state diagrams 
used to control the remote procedure can (RPC) man- 
agers that handle the remote method invocation on both 
the hand-held display device 170 side and the remote 
device side. Taken together, state diagrams in Figures 
21a and 21b disclose a handshake used between the 
hand-heid display device 170 and an Intelligent remote 
device 150 to ensure reliable communications. 
[0151] Figure 21a Illustrates a state machine for the 
remote procedure call manager 309 in the hand-held 
display device 170 that handles the display device's side 
of a remote method invocation, in the present invention, 
the user interface program object 313 in the hand-held 
display device 170 requests remote method Invocations 
to the device driver program objects 301 located in re- 
mote intelligent devices. Referring to Figure 21a, when 
a user Interface object requests a remote method invo- 
cation, the hand-held display device's RPC manager en- 
ters the "ready to transmit" state 403. The display de- 
vice's RPC manager then sets an error countdown timer 
and transmits a packet containing a message with a 



method selector and associated data. After transmitting 
the packet, the display device's RPC manager enters a 
"waiting for reply" state 405. If the proper reply message 
is sent back, the method invocation is complete and the 

5 reply message Is analyzed at state 407. 

[0152] Remote method Invocations do not always 
complete so easily. If the device driver program object 
301 In the Intelligent remote device 150 needs more time 
to complete the method, It will send back a message 

io requesting more time. The display device's RPC man- 
ager responds by moving to state 409 where it increases 
the timeout value. If the display device's RPC manager 
does not receive a reply by the time the error countdown 
timer expires, the display device's RPC manager moves 

1* to an "error recovery" state 41 1 . If the error countdown 
timer expires less than a predetermined number of 
times, the display device's RPC manager returns to the 
"ready to transmit" state 403. If the error countdown tim- 
er reaches a predetermined limit, the display device's 

20 rpc manager abandons the remote method Invocation 
and enters a failed state 413. The user is then usually 
informed of the communications problem. 
[01 53] Figure 21 b illustrates a state machine for a re- 
mote procedure can manager 307 in an intelligent re- 

25 mote device that handles the remote device's side of a 
remote method invocation. The remote device's RPC 
manager 307 starts in a "waiting to receive" stale 451 . 
When a packet is received, the remote device's RPC 
manager analyzes the packet at state 453. If the re- 

so cefved packet contains a new message requesting a 
method invocation, the remote device's RPC manager 
gives the packet to an object that performs the method 
of the request at state 455. The remote device's RPC 
manager then transmits a result and returns to the wait- 
as ing state 451. 

[01 54] If the received packet contains a message that 
the remote device has already seen, the RPC manager 
of the Intelligent remote device moves to 6tate 457 
where the intelligent remote device determines whether 

40 the method invocation request was already completed 
or whether it Is still working on the request. If the Intelli- 
gent remote device has already completed the request, 
the subsystem retransmits the results at state 459 and 
returns to the waiting state 451 . Otherwise, if the Intel- 

45 Hgent remote device is still working on the method Invo- 
cation, the remote device's RPC manager moves to 
state 461 where the remote device's RPC manager re- 
quests more time to complete the Invoked method and 
then returns to the waiting state 451 . 

po [0155] Figure 21c illustrates the sequence of a re- 
mote method invocation where there are no communi- 
cation problems. At step 501 , a user Interface object In 
the hand-held display device requests a remote method 
invocation. Therefore, the user interface object sends a 

55 message with a method selector and argument to the 
display device's RPC manager. The display device's 
RPC manager packs the message received Into a pack- 
et at step 502 and transmits the packet across the com- 
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munlcatlons network at step 503. 
[01 56] The intelligent remote devices RPC manager 
receives the transmitted packet at step 504. The remote 
device's RPC manager unpacks the message from the 
packet at step 505 and directs the packet to the device 
driver program object within the intelligent remote de- 
vice. The device driver program object Invokes the 
method specified by the method selector In the message 
at step 506. The device driver program object generates 
a return message that is passed back to the remote de- 
vice's RPC manager. At step 507, the remote device's 
RPC manager packs the return message Into a reply 
buffer. The remote device's RPC manager sends the re- 
ply buffer at step 508. 

[0157] The display device's RPC manager receives 
the reply buffer at step 509 and unpacks the reply buffer 
to extract a return value at step 510. The display de- 
vice's RPC manager delivers the return value to the re- 
questing user interface program object that invoked the 
method in the device driver program object Finally, the 
user interface program object analyzes the return value 
at step 511. 

[01 58] As illustrated, the present invention can control 
an intelligent remote device by remotely Invoking meth- 
ods in a device driver program object located within the 
intelligent remote device, in addition, it will be appreci- 
ated by the reader that the remote device may also re- 
motely invoke methods in the display device. For exam- 
ple, the device driver object in a remote device may in- 
voke a method in a user Interface object in the display 
device to display a message. 
[0159] Many remote devices are not sophisticated 
enough to have internal control methods that can be in- 
voked by the display device. Therefore, the present in- 
vention provides other methods for controlling simple 
devices. 

Simple Device Control 

[01 60] In the present invention, a remote device is de- 
fined as a "simple device" if the remote device does not 
have a microprocessor or the remote device contains 
such a primitive processor that the display device can 
not Invoke control methods within the remote device. 
Figure 1 d illustrates, in block diagram form, the compo- 
nents In a simple remote device. To control a simple de- 
vice, the hand-heid display device of the present inven- 
tion transmits simple predefined control codes instead 
of invoking control methods within the simple remote de- 
vice. Therefore, to control a simple remote device, the 
hand-held display device must obtain both a user inter- 
face program object and a device driver program object 
from the simple remote device. 
[0161] An example of obtaining user interface pro- 
gram object and a device driver program object from a 
simple remote device is given with reference to Figures 
22a - 22d, 23a - 23d and the flowchart in Figure 24. 
Figures 22a - 22d illustrate on a high level how the re- 



mote simple device 155 and the hand-held display de- 
vice 170 of the present invention communicate with 
each other. Figures 23a - 23d illustrate, in conceptual 
form, how the user interface program object 353 and the 
5 device driver program object 351 of the remote external . 
device 150 are transported the hand-held display device 
170 and used. 

[0162] Referring to Figure 22s, a thermostat imple- 
mented as a simple remote device 155 is illustrated. As 

io specified by step 124 of Figure 24, the simple remote 
device 1 55 of Figure 22s transmits both a user interface 
program object and a device driver program object to 
the hand-held display device 170. Illustrated conceptu- 
ally In Figure 23s, the simple device 155 transmits the 

13 user interface program object 353 and the device driver 
program object 351. The transmission of the program 
objects is done using some communications means 1 62 
that may be a communications network or any other 
means of transferring the information Into the display de- 

20 vice 170. 

[0163] The hand-held display device 1 70 receives the 
copy of the user interface program object 363 that de- 
scribes the simple device's graphical user interface. The 
hand-held display device 170 then Invokes a method In 
25 the user interface program object 363 that renders a dis- 
play object within a Space shown on the hand-held dis- 
play device 1 70. When a user Selects the display object 
associated with the simple remote device 155, the dis- 
play device 170 invokes a method within the user Inter- 
so face program object 363 as stated In step 125 of Figure 
24 and illustrated In Figure 22b. The method invoked 
in the user interface program object 363 displays a 
graphical user interface associated with the simple re- 
mote device on the display screen 171 of the display 
35 device 1 70. The user can then interact with simple re- 
mote device's graphical user interface on display device 
170 as stated in step 127. 

[01 64] The user's interactions with the simple device's 
graphical user interface are analyzed by the simple de- 

40 vice's device driver program object 361 within the dis- 
play device 150. The device driver program object 361 
resolves the user's interactions into control codes that 
the simple remote device 155 will respond to. As illus- 
trated in Figure 23c, the device driver program object 

45 361 located within the display device 1 70 analyzes the 
user's actions. 

[0165] Referring to Figures 22c and 23c, the hand- 
held display device 170 transmits the control codes to 
the simple remote device 155. The simple remote de- 

50 vice 1 55 changes in response to the control codes re- 
ceived as stated in step 1 30. Finally, if the simple remote 
device 155 and the display device 170 are coupled to- 
gether by a communications network, the simple device 
155 can transmit any return values or feedback gener- 

55 ated by the simple remote device 155 as illustrated In 
Figure 22d. 
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Conventional Device Control 

[0166] Existing remote devices that are not designed 
for the hand-held display device of the present invention 
do not have a user interface program object to transmit 5 
to the hand-held display device. For example. Figure 
1e illustrates a typical existing remote device that can 
only respond to coded command signals such as Infra- 
red signals. In such cases, the hand-held display device 
must be manually Informed about the remote device. 10 
The hand-held display device then uses a generic user 
Interface program object for the type of remote device. 
[0167] For example, if a user has a conventional tel- 
evision that is controlled by Infrared signals, the user 
enters a model number of the television to inform the « 
hand-held display device about the conventional televi- 
sion. The hand-held display device then accesses a da- 
tabase to obtain a list of the control codes that the con- 
ventional television will understands. A television dis- 
play object is then created to control the conventional 20 
television. If the user selects television display object, 
the hand-held display device will invoice a method that 
displays a generic television user interface. When the 
user Interacts with the generic television user Interface, 
the display device will transmit the proper infrared con- & 
trol codes to control the television. 

Summary 

[0168] The foregoing has described an intuitive 30 
graphical user interface based upon a geographic map 
structure and a system for controlling remote external 
electronic devices. In the user Interface of the present 
Invention, each Space of the geographic map structure 
is rendered on a touch screen display as a Space. Within 35 
each Space are colored icons called "objects'' that can 
be Selected and manipulated by the user. Certain ob- 
jects, referred to as portals, transport the user from one 
Space to another Space when Selected. Other objects, 
referred to as buttons, perform associated actions or *o 
functions when Selected. The user Interface is dis- 
played on a hand-held display device that can control 
remote electronic devices. Each remote electronic de- 
vice transmits a user interface program object that de- 
fines a graphical user interface to the display device. 45 
The hand-held display device displays the graphical us- 
er Interface associated with the remote device when a 
user selects an icon associated with the remote device 
on the display device's touch screen display. It Is con- 
templated that changes and modifications may be made so 
by one of ordinary skill in the art, to the materials and 
arrangements of elements of the present invention with- 
out departing from the scope of the Invention. 
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Claims 



1 . A method of providing a graphical user interface on 



a computer system (170), said computer system 
comprising a CPU (30), a memory (33), and graph- 
ics display screen (37), said method comprising the 
steps of: 

providing a plurality of spaces (32, 50, 60 ...), 
each of said spaces comprising a graphic back- 
ground Image; 

organising said spaces (32, 50, 60 ...) in a di- 
rected graph structure, said spaces connected 
together with a pluraUty of portals (47, 48, 49, 
52,54,...); 

- locating said plurality of portals (47, 48, 49 ...) 
In said spaces (32, 50, 60...), each of said por- 
tals having an associated target space such 
that said portal connects the space said portal 
is In with said associated target space, each of 
said portals comprising a graphic icon; 
rendering a graphic background Image of a first 
space and portals located in said first space on 
said graphics display screen, said first space 
having at least a first portal; 
receiving input from a user, said input compris- 
ing a portal selected from said portals located 
in said first space; and 

rendering a graphic background image of a tar- 
get space associated with said selected portal 
on said graphics display screen (37); such that 
said user can move between spaces in said di- 
rected graph structure by selecting said portals 
within said spaces; and 
transferring selected objects in the first space 
to the target space using a second portal, said 
interface conveying state information to the us- 
er by animating a visual appearance of said 
second portal to catch the users attention. 

2. The method of providing a graphical user interface 
on a computer system (1 70) as claimed in claim 1 , 
further wherein said steps of rendering a back- 
ground image further comprises the steps of: 

rendering a first subset of said background Im- 
age on said graphics display screen (37), said 
first subset from a first location in said back- 
ground image; 

receiving directional input from said user, said 
directional input comprising a movement direc- 
tion; and 

rendering a second subset of said background 
image on said graphics display screen (37), 
said second subset from a second location in 
said background image, said second location 
being said first location plus an offset corre- 
sponding to said movement direction; 
such that such user can move about within a 
background Image associated with a space. 
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The method of providing a graphical user Interface 
on a computer system (1 70) as claimed In claim 2 
further wherein said step of receiving directional In* 
put from said user further comprises detecting a fin- 
ger of said user sliding across said display screen 
(37). 

The method of providing a graphical user interface 
on a computer system (170) as claimed In claim 1 
further comprising the steps of: 

providing a wayback portal (45), said wayback 
portal displayed In each of said spaces (32, 50, 
60...). said wayback portal (45) comprising a 
stack of data structures, each of said data 
structures comprising a pointer, each of said 
pointers pointing to one of said spaces; 
pushing a first data structure on said wayback 
portal stack when said user moves to said first 
space, said first data structure containing a 
pointer to said first space; 
popping a second data structure off of said way- 
back portal stack when said user selects said 
wayback portal, said second data structure 
containing a pointer to a second space; and 
moving said user to said second space pointed 
to by said pointer in said second data structure 
popped off said wayback portal stack; 
such that said user can trace back along a path 
of spaces which said user travels through. 

The method of providing a graphical user interface 
on a computer system (1 70) as claimed in claim 4, 
wherein said data structures in said wayback portal 
(45) further comprises a coordinate, said coordinate 
Identifying a location in said first space where said 
user was when said user left said first space. 

The method of providing a graphical user Interface 
on a computer system (1 70) as claimed in claim 4, 
wherein said wayback portal (45) is rendered as a 
snapshot of a space pointed to by a pointer in a data 
structure on top of said wayback portal stack. 

The method of providing a graphical user interface 
on a computer system (170) as claimed in claim 1 . 
further comprising the steps of: 

providing a plurality of button objects (61, 62, 
66, 67, 68, 69), each of said button objects as- 
sociated with a function; 
rendering said button objects within said spac- 
es, said button objects comprising graphic 
icons; and 

executing the function associated with a button 
object when said button object Is selected by a 
user. 
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8. The method of providing a graphical user interface 
on a computer system (1 70) as claimed in claim 1 , 
wherein said display screen (37) comprises a touch 
sensitive display screen. 

9. The method of providing a graphical user Interface 
on a computer system (1 70) as claimed in claim 8 
wherein said input from said user comprises a finger 
touching said touch sensitive display screen (37). 

10. The method of providing a graphical user interface 
on a corrputer system (1 70) as claimed In claim 1 , 
said method further comprising the steps of: 

rendering an anthropomorphic graphic charac- 
ter on said display; 

attracting said user's attention by animating 
said anthropomorphic graphic character; and 
conveying state Information to said user by 
changing a visual appearance of said anthro- 
pomorphic character. 



PatentansprOche 

1 . Ein Verfahren zum Bereitstellen einer graphischen 
Benutzerschnittsteiie auf einem Computersystem 
(170), wobei das Computersystem eine CPU (30), 
elnen Speicher (33) und elnen Graphlkanzelgebfld- 
schirm (37) aufweist, wobel das Verfahren die 
Schritte umfaSt: 

Bereitstellen elner Mehrzahl von Raumen (32, 
50, 60 ...), wobel Jeder der Raume ein graphi- 
sches Hintergrundbild aufweist; 
Organtsieren der Raume (32, 50, 60 ...) In elner 
gerichteten Graphenstruktur, wobei die Raume 
miteinander Qber eine Mehrzahl von Portalen 
(47, 48, 49, 52, 54, ...) verbunden slnd; 
Lokalisieren der Mehrzahl von Portalen (47, 48, 
49 ...) in den Raumen (32, 50, 60 ...) f wobei Je- 
des der Portale elnen zugehdrtgen Zielraum 
derail aufweist, daB das Portal den Raum, In 
dam es sich beflndet, mit dem zugehMgen 
Zielraum verbindet, wobei Jedes der Portale ein 
graphlsches Icon aufweist; 
Wiedergeben eines graphischen Hintergrund- 
bilds, eines ersten Raums und eines in dem er- 
sten Raum angeordneten Portals auf dem Gra- 
phikanzeigebilGlschirm, wobei der erste Raum 
wenlgstens ein erstes Portal aufweist; 
Empfangen einer Eingabe von einem Benutzer, 
wobei die Eingabe ein a us den in dem ersten 
Raum angeordneten Portalen ausgewfihltes 
Portal umfaBt; und 

Wiedergeben eines graphischen Hintergrund- 
bildes eines dem ausgewahlten Portal zuge- 
ordneten Zlelraums auf dem Graphlkanzelge- 



22 



43 



EP0626 635B1 



44 



bikfcchlrm (37); so daB sich der Benutzer zwl- 
schen den Rfiumen in der gerichteten Gra- 
phenstruktur bewegen kann, indem er die Por- 
tale in den Rfiumen auswfihlt; und 
Ubertragen ausgewfihlter Objekte In dem er- 5 
sten Raum in den Zielraum unter Verwendung 
eines zwelten Portals, wobei die Schnlttstelle 
Zustandsinformatlonen an den Benutzer Ober- 
mittelt, indem sie ein visuelles Erscheinungs- 
bild des zwelten Portals anlmlert, urn die Auf- io 
merksamkeit des Benutzers darauf zu lenken. 

2. Das Verfahren zum Bereitstellen einer graphischen 
Benutzerschnlttstelle auf elnem Computersystem 
(170) nach Anspruch 1, wobei femer die Schritte « 
des Wledergebens eines Hintergrundbildes die 
Schritte umfassen: 

Wiedergeben elner ersten Teilmenge des Hin- 
tergrundbildes auf dem Graphlkanzelgeblld- 
schirm (37), wobei die erste Teilmenge a us ei- 
nem ersten Ort In dem Hlntergrundbild herrflhrt; 
Empfangen einer Richtungseingabe von dem 
Benutzer, wobei die Richtungseingabe elne 
Bewegungsrichtung umfaBt; und & 
Wiedergeben einer zwelten Teilmenge des Hin- 
tergrundbildes auf dem Graphikanzelgebild- 
schirm (37), wobei die zweite Teilmenge aus el- 
nem zwelten Ort in dem Hintergrundbild her- 
rflhrt, wobei der zweite Ort dem ersten Ort plus & 
elnem der Bewegungsrichtung entsprechen- 
den Versatz entspricht; 

so daB dleser Benutzer sich in einem Hinter- 
grundbikJ, der elnem Raum zugeordnet 1st, be- 
wegen kann. & 

3. Das Verfahren zum Bereitstellen einer graphischen 
Benutzerschnlttstelle auf elnem Computersystem 
(1 70) nach Anspruch 2, wobei femer der Schritt des 
Empfangens einer Richtungseingabe von dem Be- 40 
nutzer das Erf assen des Gleltens eines Fingers des 
Benutzers Qber den Anzeigebildschirm (37) um- 
faBt. 

4. Das Verfahren zum Bereitstellen einer graphischen 45 
Benutzerschnlttstelle auf elnem Computersystem 
(170) nach Anspruch 1, femer umfassend die 
Schritte: 

Bereitstellen eines ROckwegportals (45), wobei so 
das ROckwegportal in Jedem der RAume (32, 
50, 60 ...) angezelgt wird, wobei das ROckweg- 
portal (45) elnen Stapel von Datenstrukturen 
aufweist, wobei jede der Datenstrukturen einen 
Zelger aufweist, wobei Jeder der Zelger auf el- & 
nen der RAume verweist; 
Schieben einer ersten Datenstruktur auf den 
ROckwegportaistapel, wenn sich der Benutzer 



in den ersten Raum bewegt, wobei die erste 
Datenstruktur elnen Zelger auf den ersten 
Raum enthfilt; 

Wegnehmen einer zwelten Datenstruktur von 
dem ROckwegportaistapel, wenn der Benutzer 
das ROckwegportal auswfihlt, wobei die zweite 
Datenstruktur einen Zelger auf elnen zwelten 
Raum enthfilt; und 

Bewegen des Benutzers in den zwelten Raum, 
auf den durch den Zelger in der zwelten Daten- 
struktur, die von dem ROckwegportaistapel 
weggenommen worden 1st, verwtesen wlrd; 
so daB der Benutzer entiang eines Pfedes von 
RAumen zurOckgefOhrt werden kann, durch die 
sich der Benutzer bewegt. 

5. Das Verfahren zum Bereitstellen einer graphischen 
Benutzerschnlttstelle auf elnem Computersystem 
(170) nach Anspruch 4, wobei die Datenstrukturen 
In dem ROckwegportal (45) femer elne Koordinate 
aufweisen, wobei die Koordinate einen Ort in dem 
ersten Raum ktentlfizlert, an dem der Benutzer war, 
als der Benutzer den ersten Raum verfleB. 

6. Das Verfahren zum Bereitstellen einer graphischen 
Benutzerschnlttstelle auf einem Computersystem 
(170) nach Anspruch 4, wobei das ROckwegportal 
(45) als Momentaufnahme eines Raumes, auf den 
durch einen Zelger in einer Datenstruktur an der 
Oberseite des ROckwegportatetapels verwiesen 
wlrd, wiedergegeben wlrd. 

7. Das Verfahren zum Bereitstellen einer graphischen 
Benutzerschntltstelle auf einem Cemputefsystem 
(170) nach Anspruch 1, femer umfassend die 
Schritte: 

Bereitstellen einer Mehrzahl von Schaltflfi- 
chenobjekten (61 , 62, 66, 67, 68, 69), wobei je- 
dem der Schaltflfichenobjekte elne Funktion 
zugeordnet 1st; 

Wiedergeben der Schaitfldchenobjekte In den 
Rfiumen, wobei die Schaitfldchenobjekte gra- 
ph teche Icons umfassen; und 
Ausfuhren der Funktion, die einem Schaltflfi- 
chenobjekt zugeordnet 1st, wenn das Schaltfla- 
chenobjekt von einem Benutzer ausgewfihit 
wlrd. 

8. Das Verfahren zum Bereitstellen einer graphischen 
Benutzerschnlttstelle auf einem Computersystem 
(170) nach Anspruch 1, wobei der Anzeigebild- 
schirm (37) ein berOhrungsempfindlicher Anzeige- 
bildschirm 1st 

9. Das Verfahren zum Bereitstellen elner graphischen 
Benutzerschnlttstelle auf einem Computersystem 
(1 70) nach Anspruch 8, wobei die Elngabe von dem 
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Benutzer eina FlngerberOhrung des berflhrungs- 
empflndlichen AnzelgebUdschirms (37) umfaBt 

10. Das Verfahren zum Bereitstellen einer graphlschen 
Benutzerschnlttsteile auf elnem Computersystem 5 
(1 70) nacft Anspruch 1 , wobel das Verfahren femer 
die Schritte umfaBt: 

Wiedergeben eines anthropomorphischen gra- 
phlschen Zelchens auf der Anzelge; « 
Wecken der Aufmerksamkeit des Benutzers, 
Indem das anthropomorphlsche graphlsche 
Zeichen animiert wird; und 
Gbermitteln von Zustandsinformationen an den 
Benutzer, indem das visuelle Erscheinungsbild « 
des anthropomorphischen Zelchens geandert 
wird. 



Revindication* 20 

1. Un precede pour foumir une interface utiiisateur 
graphique sur un systems informatique (170), iedit 
systeme informatique etant compose (f une unrt6 
c*ntrale(30),cfunememoire(33)ettf^^ 
ftehage graphique (37), Iedit precede comprenant 
les etapes de : 

foumlture cfune pluralite cfespaces (32, 50, 
60...), chacun desdits espaces comprenant *> 
une Image graphique cfarriere-plan ; 
organisation desdits espaces (32, 50, 60...) en 
une structure de g raphe orient e, lesdits espa- 
ces etant connectes les uns aux autres par une 3. 
pluralite de portails (47, 48, 40, 52, 54...) ; 35 
localisation de ladite pluralite de portails (47, 
48, 49...) dans lesdits espaces (32, 50, 60...), 
chacun desdits portails ayant un espace dble 
associe de telle sorts que Iedit portail connecte 
i'espace dans lequel se trouve Iedit portail audit *o 
espace cible assocte, chacun desdits portails 4. 
comprenant une bone graphique ; 
rendu cfune Image graphique cfarriere-plan 
cfun premier espace et des portails situes dans 
Iedit premier espace sur Iedit ecran cTafflchage 45 
graphique, Iedit premier espace ayant au moins 
un premier portail; 

reception cfune entree provenant cTun utiiisa- 
teur, ladite entree comprenant un portail selec- 
tionne parmi lesdits portails situes dans Iedit *> 
premier espace ;et 

rendu <fune Image graphique cfarriere-plan 
cfun espace dble assocl6 audit portail setec- 
tionne sur Iedit ecran cfafhehage graphique 
(37) ; de teHe sorte que Iedit utiiisateur puisse « 
se dep lacer entre les espaces dans ladite struc- 
ture de graphe oriente, en selectionnant lesdits 
portails au seln desdits espaces ; et 



transfert cfobjets setectionnes du premier es- 
pace vers respace cible via un deuxleme por- 
tail, ladite interface communiquant des infor- 
mations rfetat a I'utilisateur en animant une ap- 
parence visuelle dudit deuxleme portail pour at- 
tirer Pattention des utilisateurs. 

Le precede pour foumir une interface utiiisateur 
graphique sur un systeme informatique (1 70) sek>n 
la revendication 1, dans lequel lesdites etapes de 
rendu cTune Image cfarriere-plan comprennent en 
outre les etapes de : 

rendu (fun premier sous-ensemble de ladite 
image cfarriere-plan sur Iedit ecran d'afftehage 
graphique (37), Iedit premier sous-ensemble 
provenant cfun premier emplacement dans la- 
dite image cfarriere-plan ; 
reception cfune entree directionneile dudit uti- 
iisateur, ladite entree directionneile compre- 
nant une direction de emplacement ; et 
rendu cfun deuxleme sous-ensemble de ladite 
image cfarriere-plan sur Iedit ecran d'afftehage 
graphique (37), Iedit deuxleme sous-ensemble 
provenant cfun deuxleme emplacement dans 
ladite image cfarriere-plan, Iedit deuxleme em- 
placement etant Iedit premier emplacement 
plus un decaJage correspondent a ladite direc- 
tion de emplacement ; 

de telle sorte que cet utiiisateur peut se dep la- 
cer au seln de Nmage cfarriere-plan associee 
a un espace. 

Le precede pour foumir une interface utiiisateur 
graphique sur un systeme informatique (1 70) selon 
la revendication 2, dans lequel, en outre ladite eta- 
pe de reception cfune entree directionneile dudit uti- 
iisateur comporte la detection cfun doigt dudit utiii- 
sateur se deplacant sur Iedit ecran cfaffichage (37). 

Le precede pour foumir une interface utiiisateur 
graphique sur un systeme informatique (1 70) selon 
la revendication 1 , comprenant en outre les etapes 
de: 

foumlture d'un portail de retour (45), Iedit portail 
de retour est affichb dans chacun desdits es- 
paces (32, 50, 60...). Iedit portail de retour (45) 
comprenant une pile de structures de donnees, 
chacun e desdites structures de donnees com- 
prenant un pointeur, chacun desdits pointeurs 
pointant sur un desdits espaces ; 
empilage cfune premiere structure de donnees 
sur ladite pUe de portail de retour lorsque Iedit 
utiiisateur se deplace vers Iedit premier espa- 
ce, ladite premiere structure de donnees con- 
tenant un pointeur vers Iedit premier espace ; 
extraction cfune deuxleme structure de don- 
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nees de iadite pile do portail do retour lorsquo 
ledit utifisateur selectionne I edit portail de re- 
tour, Iadite deuxleme structure de donnees 
contenant un pointeur vers un deuxleme 
espace ; et 5 
deplacement ducfit utillsateur vers ledit deuxle- 
me espace sur lequei polnte ledit pointeur dans 
Iadite deuxleme structure de donnees extralte 
de Iadite pile de portail de retour ; 
de telle sorte que ledit utiHsateur pout retracer io 
un itindraire des espaces traverses par ledit uti- 
llsateur. 

5. Le procedd pour foumir une interface utillsateur 
graphlque sur un systeme Inf ormatlque (1 70) selon « 
la revendication 4, dans lequei lesdites structures 

de donnees dans ledit portail de retour (45) com- 
prend en outre des coordonnees, lesdites coordon- 
nees Identlflant un emplacement dans ledit premier 
espace ou ledit utillsateur se trouvaft lorsque ledit 20 
utillsateur a quitte (edit premier espace. 

6. Le proc6d6 pour foumir une interface utillsateur 
graphlque sur un systeme inf ormatlque (1 70) selon 

la revendication 4, dans lequei ledit portail de retour 2s 
(45) est rendu sous la forme d*un Instantan6 <fun 
espace sur lequei polnte un pointeur dans une 
structure de donnees au sommet de Iadite pile de 
portail de retour. 

30 

7. Le proceed pour foumir une interface utillsateur 
graphlque sur un systeme inf ormatlque (1 70) selon 
la revendication 1 , comprenant en outre tes Stapes 
de: 

35 

foumiture cfune plurality oTobjets bouton (61, 
62, 66, 67, 68, 69), chacun desdits objets bou- 
ton 6tant ass ode a une f onction ; 
rendu desdits objets bouton dans lesdits espa- 
ces, lesdits objets bouton comprenant des ic6- 40 
nes graphfques ; et 

execution de la fonction associeo a un objet 
bouton lorsque ledit objet bouton est selection- 
ne par un utillsateur. 

45 

8. Le procede pour foumir une interface utillsateur 
graphlque sur un systeme inf ormatlque (1 70) selon 
la revendication 1 , dans lequei ledit ecran d'afficha- 
ge graphlque (37) comporte un ecran tactile. 

50 

9. Le procedd pour foumir une Interface utillsateur 
graphlque sur un systeme inf ormatlque (1 70) selon 
la revendication 8, dans lequei Iadite entree dudit 
utillsateur comporte un doigt touchant ledit ecran 
tactile (37). 55 

10. Le procede pour foumir une interface utilisateur 
graphlque sur un systeme inf ormatlque (1 70) selon 



la revendication 1, ledit procede comprenant en 
outre les etapes de : 

rendu (fun person nage graphlque anthropo- 
morphique sur ledit ecran ; 
animation dudit person nage anthropomorphic 
que pour attJrer rattentJon dudit utillsateur : et 
communication cf informations d'etat audit utHi- 
sateur en modifiant une apparence visuelle du- 
dit personnage sjrthropomorphique. 
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Murder,S he Spoke | ( 

Wkile Jessico (Angela lansbury)is recording one 
of her novel* for the Mystery Books for the Blind 
series, a murder occur j With Michael Cole, 
Charlie Daniels and Jonna Lee ♦ 

OrAhour. 
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